Huge methodological challenges and opportunities

Evaluating the HWCF commitment presents unique opportunities and challenges for documenting the public health impacts of concerted industry-led changes in the food supply. The HWCF companies have committed to a reduction in total calories. However, these companies represent only a portion of the processed food manufacturing sector, and only one source of foods and beverages consumed by children and adolescents.

The US food system is complex and fragmented. There are a vast number of unique foods and beverages (possibly more than 1.5 to 2 million unique Universal Product Codes in a year). There is a vast array of options for obtaining food, from street-level vending, open markets and concession stands to restaurants, fast serve and buffet-type eateries to retailers (supermarkets, big box stores, corner stores, drug stores, gas stations, etc.) and schools. There also are myriad influences on what an individual purchases, consumes and wastes. There is no one surveillance system that tracks calories, nutrients, ingredients or products from the food production level (farm, fishery, factory, or laboratory) through the supply chain to the ultimate child or adolescent consumer.

From the food supply side, this evaluation provides a start at tracking calories purchased from some key sources (grocery, drug, convenience stores, and mass merchandisers), but will miss out on other important sources such as restaurant and away-from-home sources.

From the food consumption side, tracking caloric intake changes and other factors influencing youth diets using existing datasets will be a challenge. At a minimum, it will take until 2014 (due to lags in access to NHANES 2011-2012) to track the effects of HWCF changes on youth caloric intake, and to assess the significance of such changes for childhood overweight and obesity. A major part of this challenge will be tracking changes for lower-income and race/ethnic subpopulations at greatest risk for childhood obesity.

The HWCF commitment is a “natural experiment”. The quasi-experimental research design required assumes that, in the absence of HWCF efforts, caloric intake and purchases would not change. Unfortunately, this underlying assumption is unlikely to be valid, presenting a number of analytic challenges. Understanding and documenting the public health impact of HWCF’s efforts will require careful monitoring of changes in other segments of the food system and policy environment that may affect caloric intake and purchases (changes in school, community state and national food and related policies such as menu labeling or soda taxes or economic recovery stimuli).

It will not be possible to fully account for shifts in many exogenous or moderating factors linked with HWCF company behaviors (and possibly those of caterers, retailers and others not part of the HWCF). For local- or state-level ecological changes (e.g., taxation and marketing regulations), there may be adequate data to control for these time-varying changes at the food purchase level but not at the daily caloric intake level. Research teams at the University of Illinois-Chicago (UIC) and other advertising monitoring efforts funded by RWJF can help provide an understanding of other changes in food marketing and in the food environment. However, changes in calories consumed or purchased due to shifts at the national level (e.g., norms, national labeling regulations, etc.) may never be fully isolated. What is possible to account for are observed changes in the economic factors under which some of these decisions are made. The UNCFRP has created a preliminary Packaged Food Purchase and Price (PFPP) Database for the 52 major market regions in the United States using Nielsen Homescan 2000-2010. Use these links to see the preliminary documentation on the methodologies used in creating the database, and list of variables created.

A certain level of uncertainty will be inherent in this process, as there will be extreme public scrutiny of this effort. The RWJF-UNC evaluation team and the EAC can provide answers only within the full context of the data available. Nonetheless, this is one of the earliest attempts to monitor changes in the US food system, and to evaluate the potential effects of the food industry’s effort to reduce caloric consumption among Americans.