This is a quick demo of how to use js-search, nodejieba to implement Chinese search in Electron.
It’s fast, real-time, faster than any other chinese search solutions, fast like never before.
| tech | version |
|---|---|
| electron | 30.0.6 |
| nodejieba | 2.6.0 |
| js-search | 2.0.1 |
This article will walk you through a number of issues you may encounter when using npm mirrors and nodejieba in mainland China:
- nodejieba package in npmmirror.com does not exist or cannot be downloaded.
- nodejieba is unmaintained and is not supported on win11 and vs studio 2022 versions.
- nodejieba does not support typescript.
Add dependencies
1npm i js-search
2npm i nodejieba@2.6.0 --save-optional --ignore-scripts
Why does nodejieba take this approach?
Because nodejieba is written in c++ and its community is no longer active.
Its installation scripts will fail. We need to skip its scripts and compile it ourselves.
** You need to install vs studio 2022 and check Use c++ desktop development **.
Or use the following powershell command to install only the needed components:
1Invoke-WebRequest -Uri 'https://aka.ms/vs/17/release/vs_BuildTools.exe' -OutFile "$env:TEMP\vs_BuildTools.exe"
2
3& "$env:TEMP\vs_BuildTools.exe" --passive --wait --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended
Fix nodejieba
nodejieba does not support the c++ 17 standard, and the way to fix it is simple.
You just need to replace StringUtil_latest.hpp in github.com/yanyiwu/limonp with nodejieba before it compiles.
Here’s a sample.
1const fs = require('fs');
2const path = require('path');
3const projectDir = path.dirname(path.resolve(__dirname));
4
5const patchFile = path.resolve(projectDir, 'SOME_FOLDER', 'StringUtil_latest.hpp'); // Save StringUtil.hpp to a local location such as SOME_FOLDER/StringUtil_latest.hpp
6
7const dest = path.resolve(projectDir, 'node_modules', ...'/nodejieba/deps/limonp/StringUtil.hpp'.split("/"));
8// first install nodejieba with `npm install nodejieba@2.6.0 --ignore-scripts`
9// https://github.com/yanyiwu/limonp/issues/34
10fs.copyFile(patchFile, dest, (err) => {
11 err && console.error(err) && process.exit(1);
12})
You can also choose to create a pr to the nodejieba repository.
I hope that all China’s open source software will have a good start and also a good finish.
Modify package.json
We still want nodejieba to be recognized by electron-rebuild when it is packaged.
1"scripts": {
2 "preinstall": "npm i nodejieba@2.6.0 --save-optional --ignore-scripts",
3 "build:plugin": "electron-rebuild -f",
electron-rebuild helps you do what node-gyp needs to do.
Write a tool to load nodejieba.
Copying nodejieba’s dictionary file
Assuming you are using Electron Builder, this code copies node_modules/nodejieba/dict/ to the root of the installation directory.
1"build": {
2 "extraFiles": [
3 {
4 "from": "node_modules/nodejieba/dict/",
5 "to": "dict/"
6 }
7 ],
Do not change any of the following lines of code.
The tool to load a local node addon
1import fs from "fs";
2import path from "path";
3import * as process from "process";
4import bindings from "bindings";
5// eslint-disable-next-line import/no-extraneous-dependencies
6import logger from "_main/logger";
7import nconsole from "_rutils/nconsole";
8import { dev } from '_utils/node-env';
9
10function loadAddon(pluginName: string) {
11 logger.log("preloading plugin");
12 let moduleRoot = path.dirname(process.execPath);
13 let tries = [["module_root", "bindings"]];
14 if (dev) {
15 moduleRoot = process.cwd();
16 tries = [["module_root", "build", "bindings"]];
17 if (!fs.existsSync(path.join(moduleRoot, "build", pluginName + ".node"))) {
18 tries = [["module_root", "bindings"]];
19 }
20 }
21 logger.log("using tries: " + JSON.stringify(tries));
22 let nodeAddon;
23 try {
24 nodeAddon = bindings({
25 bindings: pluginName,
26 module_root: moduleRoot,
27 try: tries,
28 });
29 } catch (e) {
30 logger.error(e);
31 }
32 return nodeAddon;
33}
34
35export default loadAddon;
Load nodejieba
1import path from "path";
2import loadAddon from './load_node_addon';
3
4const jbAddon = loadAddon("fastx");
5
6let dictDirRoot = process.cwd();
7if (process.env.NODE_ENV === 'development') {
8 dictDirRoot = path.resolve(process.cwd(), 'node_modules', 'nodejieba');
9}
10
11let isDictLoaded = false;
12
13const defaultDict = {
14 dict: `${dictDirRoot}/dict/jieba.dict.utf8`,
15 hmmDict: `${dictDirRoot}/dict/hmm_model.utf8`,
16 userDict: `${dictDirRoot}/dict/user.dict.utf8`,
17 idfDict: `${dictDirRoot}/dict/idf.utf8`,
18 stopWordDict: `${dictDirRoot}/dict/stop_words.utf8`,
19};
20
21interface LoadOptions {
22 dict?: string;
23 hmmDict?: string;
24 userDict?: string;
25 idfDict?: string;
26 stopWordDict?: string;
27}
28
29export const load = (dictJson?: LoadOptions) => {
30 const finalDictJson = {
31 ...defaultDict,
32 ...dictJson,
33 };
34 isDictLoaded = true;
35 return jbAddon.load(
36 finalDictJson.dict,
37 finalDictJson.hmmDict,
38 finalDictJson.userDict,
39 finalDictJson.idfDict,
40 finalDictJson.stopWordDict,
41 );
42};
43
44export const DEFAULT_DICT = defaultDict.dict;
45export const DEFAULT_HMM_DICT = defaultDict.hmmDict;
46export const DEFAULT_USER_DICT = defaultDict.userDict;
47export const DEFAULT_IDF_DICT = defaultDict.idfDict;
48export const DEFAULT_STOP_WORD_DICT = defaultDict.stopWordDict;
49
50export interface TagResult {
51 word: string;
52 tag: string;
53}
54
55export interface ExtractResult {
56 word: string;
57 weight: number;
58}
59
60const mustLoadDict = (f: any, ...args: any[]):any => {
61 if (!isDictLoaded) {
62 load();
63 }
64 return f.apply(undefined, [...args]);
65};
66
67export const cut = (content: string, strict: boolean): string[] => mustLoadDict(jbAddon.cut, content, strict);
68export const cutAll = (content: string): string[] => mustLoadDict(jbAddon.cutAll, content);
69export const cutForSearch = (content: string, strict: boolean): string[] => mustLoadDict(jbAddon.cutForSearch, content, strict);
70export const cutHMM = (content: string): string[] => mustLoadDict(jbAddon.cutHMM, content);
71export const cutSmall = (content: string, limit: number): string[] => mustLoadDict(jbAddon.cutSmall, content, limit);
72export const extract = (content: string, threshold: number): ExtractResult[] => mustLoadDict(jbAddon.extract, content, threshold);
73export const textRankExtract = (content: string, threshold: number): ExtractResult[] => mustLoadDict(jbAddon.textRankExtract, content, threshold);
74export const insertWord = (word: string): boolean => mustLoadDict(jbAddon.insertWord, word);
75export const tag = (content: string): TagResult[] => mustLoadDict(jbAddon.tag, content);
76
77export default {
78 load,
79 cut,
80 cutAll,
81 cutForSearch,
82 cutHMM,
83 cutSmall,
84 extract,
85 textRankExtract,
86 insertWord,
87 tag,
88 DEFAULT_DICT,
89 DEFAULT_HMM_DICT,
90 DEFAULT_USER_DICT,
91 DEFAULT_IDF_DICT,
92 DEFAULT_STOP_WORD_DICT,
93};
You should expose the tool, through the global object like window, to the renderer process, which can then call methods such as window.myAddons.cutForSearch.
Combine js-search and nodejieba
Assuming you want to search an object like this:
1export interface Product {
2 [key: string]: any;
3
4 productCode: string;
5 name: string;
6 namePinyin: string;
7 nameEnglish: string;
8}
Write the code in your search component like this:
1import * as JsSearch from 'js-search';
2import { Search } from 'js-search';
3
4const [search, setSearch] = React.useState<string>("");
5const jsSearchGames = React.useRef<Search>();
6const [omnisearch_games, setOmnisearchGames] = React.useState<any[]>([]);
7const [omnisearch_loading, setOmnisearchLoading] = React.useState(false);
8
9// ...
10
11// construct search component and data on load
12useEffect(() => {
13 const buildJsSearch = (uidField: string, documents: any[], ...index: string[]) => {
14 const jsSearch = new JsSearch.Search(uidField);
15 jsSearch.tokenizer = {
16 tokenize: (text) => {
17 const r = window.myAddons.cutForSearch(text, true); // cutForSearch is the method in the tool
18 return r;
19 },
20 };
21 index.forEach((i) => jsSearch.addIndex(i));
22 jsSearch.addDocuments(documents);
23 return jsSearch;
24 };
25
26
27 jsSearchGames.current = buildJsSearch('productCode', p, 'productCode', 'name', 'namePinyin', 'nameEnglish');
28}, []);
29
30// start to search if use type something in the search input
31useEffect((): (() => void) | void => {
32 if (!search) {
33 return;
34 }
35 const q = search.trim();
36 if (!q) {
37 return;
38 }
39 setOmnisearchGames([]);
40 setOmnisearchLoading(true);
41 // cancel last search if duration between this search and last search is less than 1 second
42 // this is something you need to consider while user using chinese input method
43 if (currentSearchId.current) {
44 clearTimeout(currentSearchId.current);
45 }
46 const doSearch = async () => new Promise<searchResult>((resolve, reject) => {
47 // start search after 1 second
48 currentSearchId.current = setTimeout(() => {
49 const result = {
50 sitemap: match_sitemap(q),
51 games: jsSearchGames.current?.search(q) as Product[],
52 gamesPrecisely: jsSearchGamesPrecisely.current?.search(q) as Product[],
53 orders: jsSearchOrders.current?.search(q) as Order[],
54 news: jsSearchNews.current?.search(q) as NotificationType[],
55 tags: jsSearchTags.current?.search(q) as Tags[],
56 };
57 resolve(result);
58 }, 200);
59 });
60 doSearch().then((d) => {
61 setOmnisearchGames(d.games.filter((p) => p.type !== Constants.API_TYPE_PRODUCT && p.type !== Constants.API_TYPE_GAMEBOX_APP));
62 if (d.games.length === 0 && q.length >= 2 && q.indexOf("'") < 0) {
63 Object.keys(requests_in_flght.current).forEach((k) => {
64 if (q.indexOf(k) === 0) {
65 clearTimeout(requests_in_flght.current[k]);
66 delete requests_in_flght.current[k];
67 }
68 });
69 // cut q to keep its largest length to 32
70 // save those record that returns empty results to server so we can improve
71 requests_in_flght.current[q] = setTimeout(() => {
72 post("/saveRecord", {
73 searchString: q.substring(0, 32),
74 }).catch(() => {
75 });
76 }, 5000);
77 }
78 })
79 .catch(openError)
80 .finally(() => setOmnisearchLoading(false));
81}, [search]);
82
83
84return (
85 <div className="OmniSearch-container">
86 {inputElement()}
87 {(search_focus || omniMouseOver || null) && search && (
88 <aside className="OmniSearch-results-container">
89 {(omnisearch_loading || null) && <div className="loading">Loading</div>}
90 {((!omnisearch_loading && omnisearch_result_count === 0) || null) && (
91 <div className="no-results">
92 Not Found
93 </div>
94 )}
95 {(omnisearch_games.length || null) && (
96 <div className="results">
97 <h3>Games</h3>
98 {omnisearch_games.map((e) => (
99 <div className="result" key={e.productCode}>
100 <Link to={`/productDetail/${e.type}/${e.productCode}`}>{e.name}</Link>
101 </div>
102 ))}
103 </div>
104 )}
105 </aside>
106 )}
107 </div>
108);
Done
Well, now you can achieve such a search effect as below.
