FIX: Enhance JSON extraction prompt to ensure processed_resume creation #431
Conversation
WalkthroughThe changes update the structured resume extraction prompt to enforce stricter, more explicit rules for JSON extraction, emphasizing precision and prohibiting assumptions or reclassification. Additionally, the resume service now injects a newly generated UUID into the extracted JSON output before schema validation. No changes were made to public interfaces. Changes
Estimated code review effort2 (~15 minutes) Suggested reviewers
Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
apps/backend/app/prompt/structured_resume.py(2 hunks)apps/backend/app/services/resume_service.py(1 hunks)
🔇 Additional comments (3)
apps/backend/app/prompt/structured_resume.py (3)
2-3: Aggressive tone may be effective for AI compliance.The threatening language ("deactivation and replacement", "critical failure") is quite aggressive but may be effective in ensuring strict compliance with extraction rules. This approach aligns well with the PR objective to reduce hallucinations and improve consistency.
4-13: Comprehensive rules address the core validation issues.The enumerated rules effectively target the specific problems mentioned in the PR objectives:
- Requiring all schema fields prevents validation errors from missing fields
- Prohibiting inference/fabrication reduces hallucinations
- Clear guidelines on achievements and section classification prevent incorrect data mapping
- Date formatting and output format specifications maintain consistency
These changes should significantly improve extraction reliability and prevent the
processed_resumeobject from becoming undefined.
25-25: Clear final instruction reinforces schema compliance.The explicit note about valid JSON conforming to the schema provides a strong final reminder of the requirements, reinforcing the stricter approach throughout the prompt.
| @@ -170,6 +170,7 @@ async def _extract_structured_json( | |||
| ) | |||
| logger.info(f"Structured Resume Prompt: {prompt}") | |||
| raw_output = await self.json_agent_manager.run(prompt=prompt) | |||
| raw_output['UUID'] = str(uuid.uuid4()) | |||
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
UUID injection implementation looks correct, but verify schema compatibility.
The implementation correctly generates a unique UUID and injects it into the raw output before validation. This addresses the PR objective of switching from AI-generated to manually generated UUIDs to prevent collisions.
However, ensure that the UUID field is defined in the StructuredResumeModel schema, otherwise this addition could cause validation errors.
🏁 Script executed:
#!/bin/bash
# Description: Verify that UUID field exists in StructuredResumeModel schema
# Expected: Find UUID field definition in the schema
# Search for StructuredResumeModel definition and UUID field
ast-grep --pattern 'class StructuredResumeModel($_):
$$$'
# Also search for UUID field in schema files
rg -A 3 -B 3 "UUID.*:" --type pyLength of output: 2187
Add UUID field to StructuredResumeModel schema
Injecting a unique UUID into raw_output is correct, but the Pydantic model in apps/backend/app/schemas/pydantic/structured_resume.py doesn’t declare a UUID field. Pydantic will silently ignore unknown keys, so your generated UUID won’t be validated or included in the model.
Please update StructuredResumeModel to include:
class StructuredResumeModel(BaseModel):
+ uuid: str = Field(..., alias="UUID")
personal_data: PersonalData = Field(..., alias="Personal Data")
experiences: List[Experience] = Field(..., alias="Experiences")
projects: List[Project] = Field(..., alias="Projects")
skills: List[Skill] = Field(..., alias="Skills")
research_work: List[ResearchWork] = Field(
default_factory=list, alias="Research Work"
)
achievements: List[str] = Field(default_factory=list, alias="Achievements")
education: List[Education] = Field(..., alias="Education")
extracted_keywords: List[str] = Field(
default_factory=list, alias="Extracted Keywords"
)This ensures the injected UUID is properly validated, serialized, and accessible on the model.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| raw_output['UUID'] = str(uuid.uuid4()) | |
| # apps/backend/app/schemas/pydantic/structured_resume.py | |
| class StructuredResumeModel(BaseModel): | |
| uuid: str = Field(..., alias="UUID") | |
| personal_data: PersonalData = Field(..., alias="Personal Data") | |
| experiences: List[Experience] = Field(..., alias="Experiences") | |
| projects: List[Project] = Field(..., alias="Projects") | |
| skills: List[Skill] = Field(..., alias="Skills") | |
| research_work: List[ResearchWork] = Field( | |
| default_factory=list, alias="Research Work" | |
| ) | |
| achievements: List[str] = Field(default_factory=list, alias="Achievements") | |
| education: List[Education] = Field(..., alias="Education") | |
| extracted_keywords: List[str] = Field( | |
| default_factory=list, alias="Extracted Keywords" | |
| ) |
🤖 Prompt for AI Agents
In apps/backend/app/services/resume_service.py at line 173, you add a 'UUID' key
to raw_output, but the StructuredResumeModel in
apps/backend/app/schemas/pydantic/structured_resume.py does not declare a UUID
field. To fix this, update the StructuredResumeModel schema by adding a UUID
field of type str or UUID, ensuring the model validates and includes this field
properly during serialization and access.
|
I made the changes that you and the bot suggested. line 1: import os While this prompt is a big improvement, the LLM can still output wrong/no data. Additionally, the personal_data can't be null. |
Pull Request Title
Some users (myself included) encountered errors when using the "Improve" feature due to missing fields in the generated JSON. This led to the
processed_resumeobject being undefined because the schema validation failed. The root cause is that the AI sometimes omits fields of the schema if doesn't found them in the resume.This PR refines the extraction prompt to enforce schema completeness and reduce duplication. It also opts for manually generating UUIDs instead of relying on the AI, which can lead to collisions or inconsistencies.
Additionally, while automatic extraction is powerful, AI is not flawless. A future improvement could include a form-based interface for reviewing and updating extracted data.
Related Issue
#430
#421
#386
Description
To improve extraction consistency, the extraction prompt was updated to use a more directive tone and include failure framing. This is a known prompt-engineering technique (role-based prompting + consequential framing) that reduces hallucinations and increases compliance with strict tasks like schema-based JSON output. Soft language like “please” was also removed, as it tends to relax constraints and allow for interpretation, which is not desired in this context.
Type
Proposed Changes
Screenshots / Code Snippets (if applicable)
Missing fields in output schema prevented the creation of
processed_resumeHow to Test
Validation error: 3 validation errors for StructuredResumeModelChecklist
Additional Information
Summary by CodeRabbit
New Features
Documentation