Can zero-shot AI find what buyers value?
A peer-reviewed study turns 12,900 Amazon smartwatch reviews into product priorities without task-specific training. A 200-review human check supports the sentiment labels, but the attribute taxonomy and strategic conclusions were not independently validated.
The short version
- What happened
- The pipeline analysed more than 12,900 Amazon smartwatch reviews collected from July to December 2024, using keyword extraction, several language models, BART-MNLI classification and importance–performance mapping.
- Who it affects
- Faster review analysis could help product teams find recurring usability and battery complaints that deserve investigation.
- What remains unproven
- The evidence comes from one product category, one retailer and reviews collected between July and December 2024.
Keep in mind
The evidence comes from one product category, one retailer and reviews collected between July and December 2024.
Only 200 reviews received a human sentiment check; confidence intervals and pre-consensus annotator agreement were not reported.
The study did not validate the review-to-attribute assignments against a labelled reference set.
Attribute frequency was treated as importance, which may not match willingness to pay, purchase decisions or business value.
The pipeline includes human screening and model-selection decisions, so it is not a fully automatic or model-independent method.
Transfer to other languages, retailers, products and later model versions remains untested.
Source analysis · 1 linked source. Primary analysed source date: 5 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The study builds a four-stage consumer-analysis pipeline
Researchers from Universitas Tidar, Sejong University and Kookmin University asked whether a business can organise consumer feedback without first paying for a large, product-specific labelled dataset. Their case study collected more than 12,900 Amazon smartwatch reviews through the Amazon Customer Feedback API from July to December 2024. One copy of the text was lowercased, tokenised, stripped of punctuation and stopwords, and lemmatised for keyword extraction. A raw-text copy was retained so the later classifier could use context and ordinary phrasing.
The proposed workflow is sequential rather than a single all-purpose chatbot prompt. KeyBERT first compared review and candidate-term embeddings, including one- and two-word phrases. Maximal Marginal Relevance then balanced relevance against repetition. A similarity threshold reduced 17,736 unconstrained candidates to 240. Language models filtered that list further, a zero-shot classifier mapped reviews to product attributes, another zero-shot step labelled sentiment, and an importance–performance analysis combined how often an attribute appeared with the average polarity of the reviews assigned to it. That separation makes the design inspectable, although errors in an early stage can still travel into every later result.[1]
Several models agreed on the broad vocabulary but not the exact list
The authors compared Llama 3.3 70B, GPT-5 nano, Gemini 2.5 Flash and DeepSeek R1 T2 as filters for the 240 candidate terms. Their retained lists ranged from 28 keywords for Llama to 75 for DeepSeek, with repeated concepts such as battery, display, strap, GPS, connectivity, notifications and waterproofing. The team selected the broader DeepSeek output so less frequent attributes would not be discarded before classification. Two human reviewers also screened the candidate pool to remove irrelevant terms and merge overlapping expressions.
A prompt then consolidated the vocabulary into eight categories: health monitoring, connectivity, display technology, design and comfort, software and smart features, activity and location tracking, power and battery management, and general functionality. For large-scale assignment, the study used the open facebook/bart-large-mnli natural-language-inference model. Each review and the eight category labels were supplied to the Hugging Face zero-shot pipeline, and the highest-scoring label became that review's product attribute. This is a reproducible classification rule, but the paper does not report a labelled evaluation of whether those assignments were correct, whether multi-attribute reviews were forced into an unsuitable single category, or how often the models disagreed on the taxonomy itself.[1]
A 200-review check supports the sentiment component
The same BART-MNLI model classified each review as positive, neutral or negative. Researchers converted those classes to scores of plus one, zero and minus one. Across the complete corpus, 8,524 reviews were labelled positive, 2,311 neutral and 2,087 negative, reported as 65.8%, 17.8% and 16.4%. To test this stage, two independent annotators labelled a random sample of 200 reviews covering different products, brands and expressions; disagreements were resolved by discussion to create a consensus reference label.
Against that reference, the automated sentiment labels achieved 0.88 overall accuracy and a macro-averaged F1 score of 0.86. Positive reviews produced precision of 0.91, recall of 0.93 and F1 of 0.92. Neutral reviews were harder, with precision of 0.82, recall of 0.79 and F1 of 0.80; negative reviews reached 0.88, 0.86 and 0.87. The check is useful because it measures the chosen model on the actual corpus instead of citing performance from an unrelated benchmark. It remains modest: 200 examples are about 1.6% of the dataset, the consensus process removes an independent measure of ambiguous cases, and the paper does not report confidence intervals or annotator agreement before reconciliation.[1]
The business map measures attention and sentiment, not demand
For each attribute, the study treated frequency as importance and mean sentiment as performance. That placed design and comfort, connectivity and health-monitoring features among the more prominent, positively perceived topics, while power and battery management emerged as an area of dissatisfaction. Brand-level summaries suggested Apple, Samsung and Garmin attracted much of the discussion while showing only moderate differences in sentiment. These outputs can help a product team decide which themes to investigate, especially when manually reading thousands of reviews would be slow.
But frequency is not the same as purchase importance. A feature may be mentioned because it failed, because a listing emphasised it, because one brand has many more reviews or because reviewers repeat a familiar complaint. A star rating, return, warranty claim, sales outcome or controlled choice experiment would answer a different question. Amazon reviewers are also a selected population: people who bought, used and chose to review a smartwatch, during one six-month period. The pipeline cannot establish that silent customers, prospective buyers or people in another market value the same attributes. Product managers should read the quadrants as a structured listening aid, not a causal ranking of investments or evidence that changing one feature will increase sales.[1]
Human review remains part of the method
Although described as zero-shot, the complete workflow is not hands-off automation. People selected thresholds, reviewed candidate keywords, reconciled attribute categories, chose the widest model output and created the human sentiment reference. Those decisions may be sensible, but they shape the final map and would need governance in a commercial setting. Teams should log model versions and prompts, preserve review dates, test for duplicated or manipulated reviews, and give analysts a way to inspect uncertain or multi-topic cases rather than accepting one label automatically.
The study was supported by Korean government information-technology and research grants and by Sejong University's industry–academic cooperation foundation; the authors declared no competing interests. It does not report whether any smartwatch manufacturer supplied data or influenced the analysis. Its strongest contribution is methodological: a relatively accessible pipeline can compress a large unlabelled corpus, and its sentiment component agrees well with a small human check. The paper is not evidence that any named brand leads the market, that automated review mining should replace customer research, or that the same accuracy will hold for another product category or language.[1]
What would change the assessment
Confidence would rise with an external test fixed before development: reviews from another retailer, time period, country and product category, with labelled attribute and sentiment assignments and brand-stratified error reporting. A multi-label comparator would show whether forcing one attribute per review loses important information. Publishing the collection protocol, code, prompts, model snapshots and de-identified examples would allow independent teams to reproduce the pipeline and test sensitivity to thresholds and model substitutions.
The more consequential business claim also needs outcome evidence. A prospective team could use the map to choose a product change, record the decision before launch, and compare sales, returns, support contacts or customer satisfaction against a credible control. Until that happens, the study supports zero-shot review analysis as a promising research and triage tool. It does not demonstrate that the inferred priorities are the right investments, that a 0.88 sentiment classifier is accurate enough for every decision, or that consumer preference can be reduced to mentions and polarity alone.[1]