Automated Incorporation of Yelp Restaurant Reviews into Foodborne Illness Surveillance and Response

State: UT Type: Promising Practice Year: 2016

An innovative surveillance tool was developed by Salt Lake County Health Department (SLCoHD) to improve outbreak detection in its population of approximately 1.2 million people. The tool incorporates a popular social media network Yelp and includes publicly posted reviews from this online community to augment existing surveillance of foodborne illness complaints. The primary goal was to conduct this surveillance in such a way that the number of new complaints justified the staff time and resources invested into finding them. Objectives included exploration of collaboration opportunities, completion of a pilot period to determine feasibility and effectiveness, and full implementation of the practice if results were satisfactory. Meetings were held with SLCoHD stakeholders from Epidemiology, Food Protection, and Administration, as well as two social media researchers from the University of Utah. University collaboration was not feasible due to financial constraints, but Epidemiology staff independently developed a Python computer program utilizing the Yelp API. This program was automatically run twice weekly with foodborne illness reviews from Yelp evaluated by Epidemiology staff ( and added to existing criteria for prioritizing food establishment inspections. Over nine months of pilot testing and implementing the practice, Yelp reviews accounted for a 15% increase in the total number of complaints used for surveillance. These additional complaints led to an 11% increase in the number of foodborne illness-related requests for inspection sent by Epidemiology to Food Protection, resulting in the identification and correction of critical violations at the food establishments. There was no financial investment required for this practice, and the benefits to SLCoHD were found to be worth the staff time invested.
According to the Centers for Disease Control and Prevention (CDC), approximately 1 in 6 Americans become ill with foodborne diseases each year (1). It is important for these individuals to report their illnesses and the suspected causes to their local health departments (LHDs) to enable outbreaks to be identified and responded to appropriately. The SLCoHD receives foodborne illness complaints from a variety of sources. The SLCoHD Bureau of Food Protection receives telephone complaints from the public, and a smaller number of telephone complaints are received in the SLCoHD Bureau of Epidemiology. The Utah Department of Health receives online foodborne illness complaints on its IGotSick website, which are forwarded to the LHDs where the complainants reside. The Utah Poison Control Center also forwards reports they receive that may have to do with foodborne illness. When feasible, the SLCoHD Bureau of Epidemiology follows up on these reports to obtain full exposure information. As described in a previous model practice, SLCoHD has combined these exposures from foodborne illness complaints with exposures identified during reportable disease investigations for food establishment-associated outbreak detection since 1998 (2). Currently, when two or more complainants or reported cases name the same food establishment within a fourteen day period, Epidemiology starts an investigation and sends a request for an onsite environmental health assessment to Food Protection. This allows potential outbreaks to be identified and responded to rapidly while limiting the deployment of food inspectors to those establishments most likely to actually be causing illness. The Council to Improve Foodborne Outbreak Response (CIFOR) recommends that LHD foodborne illness complaint reporting systems collect enough complaints to detect at least 21 outbreaks per 1,000 complaints (3). In order to meet this performance measure, SLCoHD sought to supplement its existing foodborne illness complaint reporting systems with publicly posted Yelp reviews that are indicative of foodborne illness. The New York City Department of Health and Mental Hygiene (DOHMH) conducted a pilot study during 2012 and 2013 using Yelp reviews to identify unreported cases of foodborne illness (4). Using a “private data feed” based on an agreement between Yelp and DOHMH, text classification programs developed through an additional agreement with Columbia University analyzed reviews and identified those most likely to be associated with foodborne illness. The program was deemed successful, and about half of the reviews identified by the algorithm were found to be consistent with foodborne illness during the previous four weeks. SLCoHD initially reached out to the University of Utah to attempt a similar approach, but ultimately developed its own innovative methodology using Python software (5) and Yelp’s application program interface (API) that allowed foodborne illness-related reviews to be identified without external agreements or financial commitments. SLCoHD serves a population of approximately 1.2 million people, and Yelp is accessible as a commercial website to all residents of Salt Lake County that have access to an internet connection. Though SLCoHD’s other complaint reporting systems perform admirably, Yelp provides another avenue for receiving foodborne illness complaints. This practice is closely aligned with the Food Safety focus area of CDC’s Winnable Battles. 1. Centers for Disease Control and Prevention. CDC 2011 Estimates: Findings. CDC Estimates of Foodborne Illness in the United States [Internet]. [cited 8th January 2014]. Available from: Salt Lake Valley Health Department. Automated Surveillance and Rapid Detection of Foodborne Illnesses. NACCHO Model Practices Database [Internet]. 2006. Available from: Council to Improve Foodborne Outbreak Response. Development of Target Ranges for Selected Performance Measures in the CIFOR Guidelines. 2009. Available from: Harrison C, Jorder M, Stern H, Stavinsky F, Reddy V, Hanson H, et al. Using Online Reviews by Restaurant Patrons to Identify Unreported Cases of Foodborne Illness – New York City, 2012-2013. MMWR. 2014;63(20):441-445. Available from: Python Software Foundation. Python Language Reference, version 3.4.0. Available from:
Food Safety
The primary goal of this practice was to identify additional foodborne illness complaints in Yelp to increase detection of outbreaks and identification of poor practices at restaurants that might lead to foodborne illness while minimizing staff time and resources. Objectives included exploration of collaboration opportunities, completion of a pilot period to determine feasibility and effectiveness, and, if warranted, automation of the search function and full implementation of the practice. While there were no specific quantitative targets set in advance, SLCoHD Bureau of Epidemiology staff responsible for foodborne illness surveillance and response were consulted for feedback throughout the pilot period and subsequent implementation. An initial meeting was held in early January 2015 to discuss the DOHMH approach and decide whether it was something SLCoHD should pursue. SLCoHD stakeholders from Epidemiology, Food Protection, and Administration were in attendance. A contact at the University of Utah with interest in social media research was identified and plans made to pursue collaboration between SLCoHD and the university. Immediately after the meeting, the Yelp API was discovered, and Epidemiology staff began exploratory development of a Python program to identify food establishments in Salt Lake County with Yelp reviews containing the term “food poisoning.” The response from the API was saved in a text file which would be compared to the API response at the next query in order to identify new reviews, since the API only returns associated facilities and does not identify reviews by date. The program was manually run twice weekly in conjunction with SLCoHD’s ongoing foodborne illness surveillance protocol. While this was a simplistic approach compared to the text classification algorithm employed by DOHMH, its successful identification of foodborne illness reviews resulted in the decision to begin a pilot period of unspecified duration to see how many complaints could be identified over time. It was decided that no follow-up would be attempted with those posting the Yelp reviews, but plausible reviews would be included in SLCoHD foodborne illness surveillance activities and in criteria for inspection requests to be sent to Food Protection. A second meeting was held in early February where the Python program was demonstrated and initial results presented, and the decision was made to continue to pilot the program while also continuing attempts to collaborate with the university. Roles and responsibilities were assigned such that Epidemiology would evaluate reviews and send inspection requests, Food Protection would conduct environmental investigations identified using the reviews, and public information officers would set up social media accounts for staff and handle communication with Yelp as needed. A week later, a conference call with two University of Utah professors was held. They expressed great interest in the project, but stated that the collaboration would be contingent on funding for a graduate student to work on the project. Ideas for applying for grants and scholarships were discussed, but ultimately it was decided that pursuing the collaboration further would not be feasible because of the financial constraints. The pilot was completed at the end of June, and the results were evaluated and found to be favorable. With approval from all stakeholders, the practice was fully implemented in July 2015. Windows Task Manager was used to run the Python program automatically on Monday and Thursday mornings each week, with links to the reviews on Yelp’s website automatically saved to a shared drive accessible to assigned Epidemiology investigators for review of identified complaints. The evaluation detailed below covers the six month pilot period from January through June and three months of implementation from July through the end of September. No media promotion or other efforts were made to publicize this practice, and there were no start-up or in-kind costs.
During the nine months of piloting and implementing the practice, 57 Yelp reviews were identified that were consistent with foodborne illness during the previous month, resulting in 4 requests for inspection sent to the Bureau of Food Protection for implicated food establishments. During this same time period, 199 foodborne illness complaints from Salt Lake County residents were received from UDOH’s IGotSick website, 149 telephone complaints from the SLCoHD Bureau of Food Protection, 15 from the Utah Poison Control Center, and 15 from the SLCoHD Bureau of Epidemiology, for a total of 378 non-Yelp complaints resulting in 37 foodborne illness-related requests for inspection. Overlap between reporting systems was minimal, with no Yelp reviews known to have also been reported elsewhere, though the absence of full last names in Yelp made matching unlikely. Overall, the inclusion of Yelp reviews resulted in a 15% increase over the number of complaints that would have been used for surveillance otherwise, and these additional complaints led to an 11% increase in the number of foodborne illness-related requests for inspection sent to Food Protection. Three of the facilities identified by including Yelp reviews received onsite environmental health assessments, uncovering a total of 17 critical violations, including improper cooling methods, food temperature violations, cross-contamination issues, hand sink issues, and dirty equipment. The fourth inspection request was declined by Food Protection due to insufficient information in the Yelp review and the low risk classification of the implicated food establishment. While no new outbreaks were confirmed based on the results of these requests for inspection, critical violations that may have jeopardized public health were identified and corrected, and the inclusion of such additional requests makes it more likely that higher numbers of outbreaks will be detected in the future. Epidemiology staff members were supportive of the practice and gave positive feedback. Detailed information on the number of reviews investigated was not collected during the pilot period, but was automatically saved in the files generated during the three months of full implementation. During this time, 632 Yelp reviews were evaluated for consistency with foodborne illness during the previous month. Of these, only 2.7%, or about 1 in 37, were confirmed. This is a far lower efficiency level than the 50% achieved by DOHMH. However, due to the Yelp API’s inability to receive queries by review date, the majority were old foodborne illness-associated reviews that took only seconds to rule out. Performed twice a week over three months, staff members investigated an average of 24 reviews per session, taking approximately 5-10 minutes to complete each time. This was deemed by staff to be a low burden effort that could reasonably be incorporated into other ongoing foodborne illness surveillance activities. The original goal of increasing foodborne illness complaint detection without unduly burdening staff was achieved.
While there was no financial investment required for this practice, several resources were necessary that may not be available at every LHD. The Python code used for this practice is specific to the Salt Lake County geographic area, so it would be necessary to have at least some initial assistance from someone proficient in Python or other similar coding languages and geographic information concepts. Furthermore, due to the Yelp API’s restriction to no more than 20 businesses returned per query, the Salt Lake County geographic area had to be subdivided into smaller geographic areas to permit full coverage. As more businesses are identified over time, these areas periodically need to be subdivided further and the code changed accordingly to capture all reported businesses, so technological support would likely be needed on an ongoing basis. While 5-10 minutes twice weekly was determined to be a low burden in this case, it may not be considered trivial for smaller LHDs with fewer resources. However, the benefits have been found to be worth the time commitment for SLCoHD, and it is anticipated that this practice will continue into the foreseeable future.
Colleague in my LHD