Marown/heissepreise
1
0
Fork 0
mirror of https://github.com/badlogic/heissepreise.git synced 2024-06-27 02:46:04 +02:00
Code Issues Packages Projects Releases Wiki Activity
605 Commits 2 Branches 0 Tags 3.1 GiB
main
Commit Graph

114 Commits

Author SHA1 Message Date
Mario Zechner
f2ef75e5c4 Remove old entries for discount only stores. Closes #102 2023-06-21 16:00:59 +02:00
Mario Zechner
b05702aff5 Hofer and MPREIS categories. 2023-06-21 15:20:28 +02:00
Mario Zechner
8419689f11 Clean-up, initial Hofer mapping. 2023-06-21 02:43:04 +02:00
Mario Zechner
303d25ccb5 Categories for Billa & Spar, infra to add catgories for other stores. 
Billa maps directly to the canonical categories. Spar uses a mapping file stores/spar-categories.json.

Each store has a generateCategoryMapping() function which is called once in analysis.js:updateData() and analysis.js:replay(). The function is responsible for

* Fetching the latest categories
* Merging them with already mapped categories
* Report new categories that haven't been mapped yet
* Report categories that have been mapped but are no longer part of the latest set of categories
* Save the merged mappings to disk

This schema might not work for all stores, in which case updateData() and replay() will use a knn approach to figure out the category for an item. See #81
 2023-06-21 01:29:00 +02:00
Mario Zechner
b0edbf88a5 Fix id of dm-de items, category of billa items. 2023-06-19 01:52:20 +02:00
Mario Zechner
6569b17da2 Remove binary encoding, web worker, clean-up. 2023-06-18 23:23:02 +02:00
Mario Zechner
d01d984706 Fix binary encoding of unit. 2023-06-18 14:45:35 +02:00
Mario Zechner
898fa44ff1 Add description to items where available, use it in search index. 2023-06-17 20:54:45 +02:00
Mario Zechner
c7537c341e Binary format optimization 4.4mb -> 3.9mb, don't store urls were not needed, use product-id instead of code-internal for spar items, 2023-06-17 01:11:21 +02:00
Mario Zechner
1d2eb5315e Fix some Hofer unit/quantity mistakes, add price per unit to frontend. 2023-06-13 10:09:52 +02:00
Mario Zechner
2d394c2f4a Sigh, runtime error. 2023-06-13 01:20:16 +02:00
Mario Zechner
031114ebca The great refactor. 2023-06-13 01:16:27 +02:00
Mario Zechner
fce2f928c7 Spar SI fixes 2023-06-12 15:31:46 +02:00
Mario Zechner
5c65236f6c Spar Slovenia. 2023-06-12 13:08:51 +02:00
Matthias Hochsteger
17f5b5add4 Fix unimarkt units 
Use 1 stk as fallback (7 articles so far).
The other 20 are using "mbe" -> wg (Waschgang).
 2023-06-06 21:19:30 +02:00
Mario Zechner
689d191769
Merge pull request #77 from tiefenb/main 
Add missing unimarkt "haushalt" category
 2023-06-06 20:50:22 +02:00
Matthias Hochsteger
e66121e2f2 Use price-per-unit field for weighted products in spar (seems most robust) 2023-06-06 20:38:48 +02:00
Mario Zechner
cb93299990 Fix sorting by name similarity, Billa deduplication. 2023-06-06 18:36:55 +02:00
Markus Tiefenbacher
f76be098c4 Add missing unimarkt "haushalt" category 
Products like pet food, soap, shampoo, etc.
 2023-06-06 16:55:22 +02:00
Mario Zechner
a5cd0f5de8 Fetch Billa data by category. Yields more items. Can be used for categorizing other store items, see categorize.js. 2023-06-06 15:15:39 +02:00
Mario Zechner
9d58e20bbd Fixes #74, fall back fetching REWEDE from GitHub pages. 2023-06-05 23:04:03 +02:00
Mario Zechner
f60436b59b More Spar unit fixing. 2023-06-05 22:06:38 +02:00
Mario Zechner
91f30e553f Don't lowercase Spar item names. 2023-06-05 21:59:35 +02:00
Mario Zechner
d30ef57d9b Fix a bunch of SPAR unit/quantity issues. See #71. 2023-06-05 21:42:57 +02:00
Mario Zechner
4f4c42126a Fix restore for Dossier data. 2023-06-05 21:14:10 +02:00
Mario Zechner
f83af199e6
Merge pull request #73 from mhochsteger/fix_spar_quantities_with_tausendertrennpunkt 
Fix quantities with "Tausendertrennpunkt" in Spar
 2023-06-05 14:45:36 +02:00
Matthias Hochsteger
11e40125f6 Fix quantities with "Tausendertrennpunkt" in Spar 
Didn't dare to ignore dots globally, as I discovered stuff like "500.000 GR" in
MPREIS data.

Fixes #71
 2023-06-05 14:42:42 +02:00
Matthias Hochsteger
505b3c75b3 fallback argument in convertUnit 
Fixes #70
 2023-06-05 14:26:42 +02:00
simmac
632c0db6b0 Fixed DM-DE queries. 
Fixes #45
 2023-06-04 23:42:15 +02:00
Matthias Hochsteger
ce13e6cc1d Units handling for Unimarkt, some clenaup 2023-06-03 15:09:29 +02:00
Matthias Hochsteger
e5ae715451 Don't convert weighted articles to 100g 
Issues:
- 100g feels wrong for some articles (the cheaper ones)
- priceHistory wasn't changed (so the shown prices were
  wrong/inconsistent with changed quantity)
 2023-06-02 23:40:13 +02:00
Mario Zechner
95b1bee3ad Fix "Kilogramm" conversion factor. 2023-06-02 21:28:53 +02:00
Mario Zechner
23f512087e Refactored and fix #55 
- `readJson()` now just checks for the file extension to decide whether to uncompress instead of taking a flag.
- moved migration logic from index.js to analysis.js:migrateToGzip
- fixed `restore()` in analysis.js
- also calling `migrateToGzip()` in replay.js
- Fix billa canonicalization for Dossier data
- Fix spar canonicalization for Dossier data and data from 2022.
 2023-06-02 18:34:14 +02:00
Mario Zechner
8bf0d65d89
Merge branch 'main' into compress-json 2023-06-02 16:56:22 +02:00
Mario Zechner
c6bbd0e03b Increased maxWidth to 150 in prettier config, formatted all the things. See #52. 2023-06-02 16:45:54 +02:00
Christian Tschugg
02bd7e5ff8 Compress raw data files on disk, fixes badlogic/heissepreise#51 2023-06-02 16:24:58 +02:00
Mario Zechner
8c202de9a6 Merge branch 'main' of https://github.com/badlogic/heissepreise 2023-06-01 14:40:36 +02:00
Mario Zechner
6b1f84cfe3 Manually merged PR #48, Penny support (only gets us 275 products) 2023-06-01 14:40:28 +02:00
Matthias Hochsteger
b8577bb507 Fix mpreis units 2023-06-01 08:25:04 +02:00
Matthias Hochsteger
4c55a085b2 Use short-description-2 in spar for quantity/unit 
Also fix round error in cases without quantity short-description
 2023-06-01 07:42:28 +02:00
Mario Zechner
2a53cbd91a Added Rewe Germany 2023-06-01 01:09:25 +02:00
Mario Zechner
1270002b99 Initial DM Germany support. 2023-05-30 19:29:33 +02:00
Markus Tiefenbacher
942c83e9a4 Merge branch 'main' of https://github.com/tiefenb/heissepreise 2023-05-29 09:55:54 +02:00
Markus Tiefenbacher
7012e246fd Dedect Unimarkt Bio Products 
should dedect about 99% of unimarkt bio products
 2023-05-29 09:55:51 +02:00
Mario Zechner
a7b645d1cd Fix crash with Dossier data. 2023-05-29 01:08:00 +02:00
Mario Zechner
46eb2aff72
Merge pull request #10 from mhochsteger/unify_units 
Unify units
 2023-05-29 01:01:34 +02:00
Markus Tiefenbacher
1fa31bdf7c Add Unimarkt Support 2023-05-29 00:08:32 +02:00
Matthias Hochsteger
dbe22ef5e6 Merge remote-tracking branch 'origin/main' into unify_units 2023-05-28 22:31:12 +02:00
Matthias Hochsteger
4685a41eaa Fix lidl unit conversion 2023-05-28 22:27:47 +02:00
Mario Zechner
35c3bbbcca
Merge pull request #29 from iantsch/ui-generic-store 
Refactor UI to generic store syntax
 2023-05-28 20:37:45 +02:00
Matthias Hochsteger
524ca40307 Convert all weighted articles to 100g/ml 2023-05-28 19:48:06 +02:00
Matthias Hochsteger
c08ad5dd0a isWeighted for billa, hofer, lidl, spar 2023-05-28 19:11:18 +02:00
Matthias Hochsteger
2a19207064 Remove debug output 2023-05-28 19:10:08 +02:00
Matthias Hochsteger
e4a438725d Unify error messages in shops 2023-05-28 18:51:17 +02:00
Matthias Hochsteger
2fa8a4149c Remove debug output 2023-05-28 18:48:13 +02:00
Matthias Hochsteger
55ed6f9287 Unify dm units (and fix price/unit for some items) 2023-05-28 18:47:21 +02:00
Matthias Hochsteger
9b6e1164b4 Unify lidl units 2023-05-28 18:37:06 +02:00
Matthias Hochsteger
e5b2b15eaa Fix mpreis prices for weighted articles, add field "isWeighted" to canonical data 2023-05-28 18:30:02 +02:00
Matthias Hochsteger
b48c27dfdf Convert mpreis units 2023-05-27 22:27:09 +02:00
Matthias Hochsteger
57a203c573 Parse unit and quantity in hofer and billa 2023-05-27 22:12:20 +02:00
Matthias Hochsteger
22f380c390 Generate quantity and unified unit for spar data 2023-05-27 22:12:20 +02:00
Mario Zechner
ce8e0fb102 Style changes, Ingest Dossier Data from xls converted to csv see https://www.dossier.at/dossiers/supermaerkte/quellen/anatomie-eines-supermarkts-die-methodik/ 2023-05-27 12:33:18 +02:00
Christian Tschugg
a886580469 Refactor UI to generic store syntax 2023-05-26 18:12:29 +02:00
Christian Tschugg
f2ffe5957d Refactor to generic store syntax 2023-05-26 12:45:30 +02:00