This past week marked the end of the coding period for GSoC'20. The week began on 17th August and ends today i.e. 23rd August. It was a smooth week in terms of work which mainly involved completing the pending tasks and sending them for code reviews. It is also the end of a wonderful three months of learning, developing, and most importantly, growing as a software developer.
As per the plan for this week, I worked on refactoring my commits for _include and _revinclude with the new changes made to support the integration tests. I was able to complete this with little issues and got my commits for the same merged on a separate include branch, which would be merged into master after the initial release of the FHIR v2 module.
Challenges Faced
There was just one small challenge that I really faced this week. It was related to the way in which Provider and User instances were being coupled to return the Practitioner resources. According to the initial implementation, all the resources for Provider and User were concatenated and being returned in a SimpleBundleProvider.
But this solution would fail if we were to include related resources as well. This is because the related resources for every resource matching the search criteria should be returned on the same page as the matched resource. If all the resources (matched as well as the included) were returned as a single bundle, there would not be a guarantee that the above situation would always happen.
This problem was resolved by implementing a TwoSearchQueryBundleProvider, which took the two bundle providers, in this case the Provider and the User bundle and returned the resources in the correct order, taking care of the fact that the matched resource and its related resource should be returned on the same page.
Tickets worked on during Week 12
No new tickets were created or worked upon this week. The work mostly revolved around tickets which were created and partially worked on in the previous weeks of the coding period. The tickets were on implementation of _include and _revinclude features for the search functionality. The master tickets for these are shared below:
The tickets are still in the In Progress state because they have been merged to a separate include branch and will be closed only after they are merged to master.
Plan for the Future
The next week is all about submitting the documentation regarding the project like updating the project wiki, preparing a video about the project and submission of the mentor evaluation.
Overall, it has been a great journey for the last few months. From selecting the organization, getting the first Pull Request merged, to getting selected for GSoC and then working extensively on the chosen project, it has certainly been one great ride. All of these were accompanied by a great community interaction and not to forget the endless support of my mentor, Ian Bacher.
I certainly feel I have grown a lot in these months and I am thankful to everyone who has helped me in the process. I have liked contributing to the OpenMRS codebase, and sort of developed a sense of responsibility towards the same. I hope to stay active in the community even after the program ends and look forward to a lot more contributions!
Have a great week! And...I am not going away anytime soon. Ciao for now!
The eleventh week of the coding period for GSoC'20 began on 10th August and ends today, i.e. 16th August. The week was a little tough as compared to the previous few weeks, particularly because there were some bugs in the codebase which were only highlighted at the time of writing the integration tests and it took some amount of deep-dive to get to know the root cause.
I mainly worked on implementing integration tests for Location and AllergyIntolerance along with the refactoring of some of my _include commits. I also figured out inconsistencies in the translators regarding how to deal with null inputs and resolved the same.
Challenges Faced
The majority of the challenges faced were in the implementation of resources. There were some fields like verificationStatus for AllergyIntolerance, which were required for the resource to be marked as a valid resource. There was also a logical error where we were not assigning any UUID while creating a new resource and upon creation, every resource had the id field as null. It took time to debug the root cause for each of these issues and come to a conclusion (after discussions with my mentor) as to what should be the correct step to mitigate the errors.
Apart from that, there was an issue with the Practitioner User and Provider Reference Translators where the reference was being returned by using the service to get the FHIR resource and then translating that to an OpenMrs object, resulting in a TransientPropertyValueException while creating a new object given the reference to an already existing Practitioner. The error was resolved by directly returning the OpenMrs object using the Dao layer.
Tickets worked on during Week 11
During the last week, I worked on implementing the integration tests for a couple of resources, namely Location and AllergyIntolerance. I also found some issues in the translators as they were not appropriately returning null or throwing an exception for the null input. Moreover, I also worked on refactoring some of my commits for _include to bring them in a state where they can be merged to master without any conflicts. But they are still in my local machine and the PR for them can only be created after the initial release of the FHIR v2 Module.
I worked on three issues this week and the JIRA tickets for them are as follows:
The next week is essentially the last week of the coding period for GSoC'20, before the student and mentor evaluations begin. I'll look to complete the refactoring of my _include and _revinclude commits so that they can be merged as soon as the module is released. I'll also discuss with my mentor regarding any documentation that has to be done from my side.
The tenth week of the coding period for GSoC'20 began on 3rd August and ends today i.e. 9th August. It was a good week in terms of work but there were some problems faced as well during the process. I managed to complete the _revinclude implementation for all the resources and started working on the integration tests for some resources.
Challenges Faced
As is always the case, bugs and blockers don't tend to leave developers. There were a couple of issues faced during the Week 10 development.
Firstly, while working on the integration tests for MedicationRequest, I found out that it was not as easy to support create, update and delete for Orders, because we didn't have a lot of information to construct the OrderContext. I discussed it with my mentor, Ian Bacher, and we decided to not provide this functinality in the initial version of the module because it didn't seem as easy as we expected it to be.
I ran into the second issue while implementing integration tests for Location resource. The POST request for creating resources using the specified JSON or XML file is not taking into account the specified "id", thus resulting in the created resource to have a "null" UUID. I have not been able to find a fix for this yet and will discuss the same with my mentor to correct the functionality. But as a weird fact, the JSON and XML "id" is picked up properly in a PUT request for updating the resource.
Tickets worked on during Week 10
The plan for this past week was to complete _revinclude feature implementation for the remaining resources and then start on the integration tests for the FHIR module. I completed _revinclude and worked on implementing integration tests for a couple of resources. Also, there was a not-so-good functionality for the DiagnosticReport resource, wherein we were originally returning all Obs instead of just those which were an Obs grouping. I worked on this issue and added a criterion to the search which restricted the results to only Obs groupings.
I worked on seven issues this week and the JIRA tickets for them are as follows:
In the next week, I'll be focussing on completing integration tests for Location resource and pick up a few more resources. I will also start working on refactoring the _include and _revinclude commits because GSoC is coming to an end and we would like to have those merged into master, soon after the release of the FHIR module. The commits for now are in my local desktop and we would want to push them to a separate tracking branch till the initial version of the module is not released.
The ninth week of the coding period for GSoC'20 began on 27th July and ends today i.e. 2nd August. This week also marked the second evaluation period for students and mentors to submit each others' evaluations for the last 4 weeks. I passed the evaluation with appreciation from the mentor along with a small suggestion on how to improve code at particular points. It makes me all the more motivated to keep improving and delivering for the final 4 weeks of the coding period.
As far as the work is concerned, it was a smooth week with no major issues faced. I was able to complete four issues, which included improving search and CRUD operations for some of the resources. The module is in an important phase right now where there is a lot of work needed on refactoring to support integration testing. That is why I had to park the original plan of completing the _revinclude implementation and instead focus on this.
Tickets worked on during Week 9
The initial plan for this week was to work on and finish the _revinclude implementation for all resources. But because the release of the FHIR v2 module is near, there were some urgent issues I had to work on which included improving the search for Service Request and fixing the implementation of CRUD methods for all resources, along with some necessary refactoring.
I worked on four issues this week including one on _revinclude and the JIRA tickets for them are as follows:
The tasks for the next week include implementing _revinclude for all the remanining resources and I hope to finish it by the end of the week. I will also have a conversation with my mentor, Ian Bacher, on what would be the next task for me for the remaning part of the coding period and I'll create tickets for that as and when I pick it up. Most probably, it should be writing integration tests for some of the resources, but I'll bring clarity on the exact task. For now, the JIRA tickets for the issues I'll work on are:
The eighth week of the coding period for GSoC'20 began on 20th July and ends today i.e. 26th July. This week also marks the end of the second phase of the coding period and the student/mentor evaluations will open tomorrow and are due by Friday i.e. 31st July. This week I worked on completing the _include feature implementation along with finding the right approach for _revinclude implementation, both of which are complete without any major blockers faced.
The week in fact went by pretty smoothly, also involving a couple of interesting conversations with my mentor, Ian Bacher. There were a few things regarding the code base that were not really known to me and the convo shed light on them, enhancing my knowledge of the same. I also found a small bug in the code while thinking about the _revinclude implementation, wherein the parameters were being added to the SearchParameterMap without being checked for null. This was also not getting caught by the unit tests, thus making it tougher to identify. I fixed the same and avoided issues that could trouble us at a later stage.
Tickets worked on during Week 8
Since the FHIR v2 module is going to be released soon, there are no additional feature commits being done on it for now and will be taken up only after the release. Therefore, my _include pull requests are still not created, though the work is complete for the tickets mentioned. I worked on five issues this week including one on _revinclude and the JIRA tickets for them are as follows:
The tasks for the next week include implementing _revinclude for all the resources and I hope to finish it by the end of the week. During week 8, I was able to finalize the approach for _revinclude and integrate it with _include which helped in avoiding a completely different implementation for the same. I'll continue on that and work on implemenation of the feature for different resources now.
The seventh week of the coding period for GSoC'20 began on 13th July and ends today i.e. 19th July. The week saw considerable progress on the _include feature implementation, with most of the resources supporting it now. There were no major issues faced during the week, apart from just one where some translators in the FHIR v2 module did not have null checks at various places, leading to unexpected NullPointerExceptions. I picked up this task and resolved the issue by adding null checks at all the vulnerable points in the translators. Also, with the first release of the FHIR module in mind, this was one of the important and critical tasks to be completed.
Tickets worked on during Week 7
Since the FHIR v2 module is going to be released in the next 2 weeks, there are no additional feature commits being done on it for now and will be taken up only after the release. Therefore, my _include pull requests are still not created, though the work is complete for the tickets mentioned. I worked on six issues and the JIRA tickets for them are as follows:
The tasks for the next week include implementing _include for Person, RelatedPerson and ServiceRequest resources. After these, I'll start working on _revinclude feature implementation, commonly known as reverse include.
It is different from _include in the sense that, _revinclude retreives all resources which refer to the searched resource whereas _include retrieves all resources which are referred to by the searched resource. I'll work on refactoring the module to support _revinclude and try to implement a small framework that makes it easier to extend the feature to all resources.
The sixth week of the coding period for GSoC'20 began on 6th July and ends today i.e. 12th July. The week went as expected and there were no major issues faced in the development process. I completed all the work that I had planned for this week. I was able to finalize the approach for implementing _include with the search queries and also completed a couple of tickets on _include for Location and Observation resources. Along with this, I also worked on some tickets in the FHIR module which were not related to my project like adding functionality for create, update, delete methods for Observation resource.
Tickets worked on during Week 6
I worked on five issues and the JIRA tickets for them are as follows:
The tasks for the next week include implementing _include for resources which have references associated with them. For now, I have created these four tickets to work on but will create JIRA tickets for other issues as and when I pick them. I aim to finish the _include implementation for most resources by the end of week 7. The JIRA tickets for issues I'll work on are:
The fourth week marked the end of the first phase of the coding period for GSoC'20. The student and mentor evaluations were due in the fifth week which commenced on 29th June and ends today i.e. 5th July. The evaluation went very smooth and I made through to the second phase of the coding period. I also got great feedback on my work from my mentor giving me a confidence boost and motivating me to perform even better in the rest of the coding period.
The week went by pretty quickly with no major blockers in the way. I was able to complete a couple of issues and raised pull requests for the same and are under review. I also started working on implementing the _include search parameter whose main aim is to also return all other resources which are related to a particular resource in the search results. In particular, all resources referenced to by the resource matching the search criteria will be included with it in its contained list of resources.
Challenges Faced
There were no challenges faced as such, though there is a particular design change I have in mind to support the _include search parameter.
While figuring out the implementation strategy for _include, I figured out that the only way we are actually accessing the translators is through the SearchQueryBundleProvider and we need to pass _include params to those translators. The way of doing that is to change the design of the SearchQueryBundleProvider and translators a bit to add a parameter for the _include. I am not really sure if this is the only way of dealing with the problem or if there's a neater way. I'll have a discussion with Ian over this in the next week and move on to the implementation.
Tickets worked on during Week 5
I worked on three issues and the JIRA tickets for them are as follows:
The tasks for the next week include having the design discussion with Ian on _include and then moving on to the implementation. I'll also start working on implementing _include for Location:partof and will create the JIRA tickets for other issues as and when I pick them. I will also work on adding functionality for create, update and deleet for the Observation resource. The JIRA tickets for issues I'll work on are:
The fourth week of the coding period for GSoC'20 was not a straightforward one. There were a few changes in the design for implementing some of the features and will be discussed in detail. The week began on 22nd June and ends today i.e. 28th June. With the end of this week, the phase 1 of the coding period is complete and the student/mentor evaluations for the same are due. As per the plan, most of the tickets and issues discussed for Phase 1 have been completed and an issue to implement _id and _lastUpdated search parameters is in progress.
Challenges Faced
There were a few challenges in the issues worked on during this week. I had a few discussions with my mentor, Ian Bacher, and we were able to resolve them by modifying the already planned implementation strategies. The issues were:
Implementing common search params:
The earlier plan was to implement the common search parameters like _id and _lastUpdated using an interceptor where we would examine the RequestDetails parameter. If any of _id or _lastUpdated were present, we would set an attribute in the RequestDetails parameter and access the same in the resource provider. This way we would not have to mention an explicit @OptionalParam in the resource providers for every search parameter.
But the reality is often disappointing. While adding functionality for the interceptor, I figured out that _id cannot be used without specifying it as an explicit @OptionalParam, maybe because of the way it has been handled in the HAPI FHIR library. Also, if we used interceptor, we would have to manually add the chain, resource type and value to the reference parameters by parsing the sent attribute in the RequestDetails parameter, as it is automatically not handled. Due to these two major issues, Ian and I decided to implement this functionality by adding an explicit @OptionalParam field for every parameter in the resource providers.
Ensure search returns distinct results:
While working on the issue to get only distinct search results, I tried to change the criteria query in the paging structure to get only unique UUIDs of the matching resources. I changed the query to return distinct UUIDs, but I figured out that the distinct keyword does not work together along with sorting. The next thing I tried was to instead use group by UUID, but that also seemed to be problematic as sorting parameters would have to be added to the group-by list in some cases, making it very inconvenient.
I solved the problem by keeping the query in a DetachedCriteria object (as a SubQuery) and separating the grouping by UUID from sorting. This way they would not impact each other's functionality and the main query could then select UUIDs from this subquery and group by UUID as well to eliminate duplicate results.
Tickets worked on during Week 4
I worked on three issues and the JIRA tickets for them are as follows:
The tasks for next week include incorporating any comments on the distinct search results PR and completing the changes for adding functionality for the common search parameters. I'll also start working on implementing advanced parameters like _include and will create the JIRA tickets for the same on the go. For now, I have created a main ticket for _include and will keep on adding sub-tickets to it. The JIRA tickets for issues I'll work on are:
The third week of the coding period for GSoC'20 was a very smooth one. The week began on 15th June and ends today i.e. 21st June. There were no major challenges faced during the week and I instead made a lot of bug fixes to the codebase of the FHIR v2 Module. I was able to resolve one of the blockers from Week 2, leaving me with no blocked issues at the moment. Also, the paging work is now almost complete (just a couple of tickets waiting to be merged). All the resources, which have a resource provider, support paging of search results now. I will now shift focus on improving search for some resources and on implementing the common search parameters.
Blockers Resolved
During week 2, I faced a major blocker while implementing paging for the Medication resource. The query for joining the Drug table to the Concept table via the DrugIngredient table was not working as expected and joining the Concept table twice leading to ambiguous column name error. Even after looking deeper into the issue and trying out several things, I could not solve the issue with that approach. So, I used DetachedCriteria to generate the query in a different format which avoided joining the DrugIngredient table to the Concept table and it worked smoothly.
Tickets worked on during Week 3
I worked on five issues related to implementing paging and certain bug fixes. The JIRA tickets for them are as follows:
With the paging work complete and the bugs fixed as well, I'll work on improving the search for the Practitioner resource. I also have a ticket to ensure that search results returned as part of a query are distinct in nature. Currently, the way we are handling names in a query returns multiple instances of the same resource because of the joins between tables. The way to resolve this is to change the paging structure a bit and store only the distinct UUIDs of the resources. I'll work on implementing this approach in week 4. Additionally, I'll work on adding the functionality for common search parameters like _id and _lastUpdated. The JIRA tickets for the these issues are as follows:
The second week of the coding period for GSoC'20 was an exciting one with a lot of work done and some challenges faced during the way. The second week began on 8th June and is about to end today, i.e. 14th June. This week, I worked on implementing paging for a few resources and the FHIR v2 module is on the verge of having paging functionality for all the resources it currently supports.
My workspace
Challenges Faced
As is always the case, bugs and blockers don't tend to leave developers. There were not many issues during the week, but there is one that is still left to be dealt with. The query to search Medication resource by the ingredient code was not working as expected and joining the Concept table twice. I validated it by putting logs and have still not been able to come up with a fix. Apart from this, there were not any major issues, though I found a major bug in the code base for FHIR v2, which I'll discuss with my mentor and rectify it in the next week.
Tickets worked on during Week 2
I worked on five issues related to implementing paging and the JIRA tickets for them are as follows:
There is still some paging work left for a couple of resources namely, FhirTask and RelatedPerson. After the paging issues, I'll continue to work on the bugs I found out in the codebase and they would take me atleast two days to resolve. And I have an issue under my belt to improve the search for the Practitioner resource. And finally, I'd deep dive into the query issue for Medication resource and try to come up with a fix. That is the plan for the next week, though completing everything looks a little uncertain. The JIRA tickets for the paging and search issues are as follows:
After a month-long community bonding period, the coding period finally commenced. I was really eager to start contributing to my project and I worked on a few tickets as per the discussions with my mentor. As the first week of the coding period for Google Summer of Code 2020 is about to end, I'd like to share my experience during the same.
This past week, I worked on implementing paging for some of the resources in the FHIR v2 module. Earlier, the search API returned all the resources matching a specified search criteria at once. The main aim of implementing paging is to limit the number of returned resources on a particular page (ex: returning 20 results on one page), along with the links for the previous and the next pages. This would make the search more efficient by retrieving results only as per the need and avoid unnecessary disc reads and writes.
Tickets worked on during Week 1
I worked on four tickets related to implementing paging and the JIRA tickets for them are as follows:
Apart from working on tickets, I also found out a few bugs in the codebase related to paging in Observation resource, Person Name handling in BaseDao and some improvements to remove hardcoding of constants in unit test classes. I'll take them up in the next week after initial discussions with my project mentor.
Challenges faced
With a lot of active development, come issues and solutions to them. I faced some minor issues while working on the tickets during week 1, which I was able to resolve myself or with the help of my mentor.
The datasets defined for testing of the resources (present in the test folder) were getting mixed with the standard dataset. The attributes of resources in both the datasets with the same uuid were mixing randomly leading to problems while testing code. I solved it by altogether skipping to load the standard dataset. Ian advised me to place a @SkipBaseSetup annotation on top of the test class and execute the INITIAL_XML_DATASET_PACKAGE_PATH which skipped the standard dataset.
The other issue that I went through was while implementing paging unit tests for AllergyIntolerance resource. The resource used GlobalPropertyService in dao layer and the translators. I faced a challenge mocking it because the getter on AllergyIntoleranceDaoImpl had access from the same package only. I resolved it by changing the access restriction to PUBLIC, though I would need to discuss this with Ian to see if that's alright.
Plan for Week 2
In the next week, I'll focus on implementing paging for the remaining resources. I will hopefully be able to complete paging for all resources by the end of the next week. I'll also look into the bugs and improvements I found out during Week 1. The JIRA tickets for the next week are:
The community bonding period for GSoC 2020 extended from May 4, 2020 to May 31, 2020. Since the four-week long community bonding is now almost complete and the coding period (Phase - I) is on the verge of commencing, I'd like to share my experience working during the same through this blog.
Reading Documentation
The community bonding period gave me an opportunity to go through the code base. This helped me in understanding all the functionality that has been implemented in the FHIR module, particularly the search part of it. Since the paging implementation had already begun by the time projects were announced, I got an overview of what all changes had been made to the code base to support it and what all will be required to be added to the resources that do not support paging currently.
I also read a lot of documentation regarding my project, including the HAPI FHIR library on search and interceptors. Since my project includes implementing Lucene queries and integrating GraphQL with the FHIR module as a stretch goal, I also got a chance to look into those aspects as well.
Coding Shouldn't Stop
On the coding side, I got my hands dirty on various issues in the FHIR module. While going through the code for FHIR search, I found some bugs and created JIRA tickets for the same and rectified them. I also worked on implementing search for some resources like DiagnosticReport which didn't have the functionality. The list of issues I worked on in the community bonding period is as follows:
An important part of the community bonding period is to plan for the work to be done in the coding period and get to know your mentors. I had several conversations with my mentor, Ian Bacher, where we used to discuss the goals and the expectations for the coding period and also got to know each other in a better way. We also talked about any issues that I faced and discussed on ways of resolving them.
We decided to create JIRA tickets for issues as and when I start working on them and hence worked out the goals for phase-I of the coding period. The two major things I'll be working on are implementing the common search parameters, including _id and _lastUpdated, and implement paging for resources which currently do not support it. The JIRA tickets for the same can be found below:
The Improve FHIR Search project is in progress and I have already started working on the paging tickets. I was supposed to work on implementing the functionality for the common search parameters first, but owing to a lot of non-uniformity in the code base because some resources have paging implemented and some don't, my mentor and I decided to first work on paging and then implement the common search parameters after the code base becomes balanced.
Project Discussion Thread: The public discussions related to my project can be found on this OpenMRS Talk thread.
It was around 22:30 Hrs (UTC +5:30), on the night of 4th May 2020, I had just had my dinner and went back to my room. There was an hour left for something that had kept me waiting and waiting for more than a month. It was the time when the Google Summer of Code results would be announced, sharp at 23:30 Hrs . I lay on my bed pondering, “what should I do for the next 60 minutes?”.
After a while of skimming through possible ideas, I figured out I was too nervous to do anything. So, I just decided to stay in that position and think of the entire process I had followed to get to this point, being patient while starting to contribute to open source even though most of the things went over my head, being courageous enough to not leave it in between, and finally to be able to press the “Submit Proposal” button on the dashboard.
I was going through mixed feelings, what would happen if I don’t get in, what would I do if I did and how would the next few months look like. I tried to not get too carried away by all this but I couldn’t really stop myself, for I had definitely put in a great amount of effort and I felt I had a great chance as well. I finally agreed to take this as a part of the experience and rejoice it while it lasted…something I’ve learnt the hard way!
So, after managing to get to 23:15 Hrs, I stood up, had some water to relax myself, made a glass of mango shake which happens to be my favourite summer drink. So, it’s finally 23:25 Hrs and I opened my laptop, staring at the projects page and my mind ticking like a clock, counting every second that passed by.
Google, being very punctual with the timeline, announced the results at sharp 23:30 Hrs. I refreshed the page, searched for my name. No results found for your search is what I got. I refreshed the page again with my heart in my mouth, and searched once more. And then my eyes glowed seeing it there with my preferred org. My happiness knew no bounds, I jumped in excitement, it was the moment I had been dreaming of for 4 months. I got an email from Google, which said: “Congratulations, your proposal with OpenMRS has been accepted!”.
I was now a member of the OpenMRS community and looked forward to contributing extensively to their code base :)
What is OpenMRS?
OpenMRS is a Medical Records System to support the delivery of health care, developed for the underprivileged sections of the society. The OpenMRS community is always ready to assist you with any issues you might be facing. The tag line, “Write Code. Save Lives.” , inspires contributing more to this wonderful organization. The community offers a great platform to learn and grow as a developer, with a very systematic and well-defined structure of almost everything from issues to wikis for getting onboarded real quick.
The Project: Improve FHIR Search
FHIR (or Fast Healthcare Interoperability Resources) is a standard for exchanging electronic health records. It describes elements (called resources) and an API (or application programming interface) for implementation of the same. The project comprises of improving the search of the FHIR module of OpenMRS by adding support for the exhaustive list of search parameters specified by the HAPI FHIR library.
The objectives of the project are as follows:
Adding implementation for the remaining search parameters for the most useful resources including Location, Observation, Encounter, DiagnosticReport, Patient, Person etc.
Adding implementation for common search parameters including _id and _lastUpdated.
Adding paging functionality to resources which do not support it currently.
Implementing the advanced search parameters like _include, _elements and _summary which let you search in a more specific manner.
Adding support for Lucene indices available for certain resources by implementing LuceneQuery instead of CriteriaQuery.
Integrating the module with GraphQL as a stretch goal.
