|
0
|
1 Nutch Change Log
|
|
|
2
|
|
|
3 (trunk): Current Development
|
|
|
4
|
|
|
5 * NUTCH-1274 Fix [cast] javac warnings (Tejas Patil via lewismc)
|
|
|
6
|
|
|
7 * NUTCH-1494 RSS feed plugin seems broken (Sourajit Basak, Tejas Patil and lewismc)
|
|
|
8
|
|
|
9 * NUTCH-1127 JUnit test for urlfilter-validator (Tejas Patil via lewismc)
|
|
|
10
|
|
|
11 * NUTCH-1119 JUnit test for index-static (Tejas Patil via lewismc)
|
|
|
12
|
|
|
13 * NUTCH-1510 Upgrade to Hadoop 1.1.1 (markus)
|
|
|
14
|
|
|
15 * NUTCH-1118 JUnit test for index-basic (Tejas Patil via lewismc)
|
|
|
16
|
|
|
17 * NUTCH-1331 limit crawler to defined depth (jnioche)
|
|
|
18
|
|
|
19 Release 1.6 - 23/11/2012
|
|
|
20
|
|
|
21 * NUTCH-1370 Expose exact number of urls injected @runtime (snagel via lewismc)
|
|
|
22
|
|
|
23 * NUTCH-1117 JUnit test for index-anchor (lewismc)
|
|
|
24
|
|
|
25 * NUTCH-1451 Upgrade automaton jar to 1.11-8 (lewismc)
|
|
|
26
|
|
|
27 * NUTCH-1488 bin/nutch to run junit from any directory (snagel via lewismc)
|
|
|
28
|
|
|
29 * NUTCH-1493 Error adding field 'contentLength'='' during solrindex using index-more (Nathan Gass via lewismc)
|
|
|
30
|
|
|
31 * NUTCH-1491 Strip UTF-8 non-character codepoints in title (Nathan Gass via markus)
|
|
|
32
|
|
|
33 * NUTCH-1421 RegexURLNormalizer to only skip rules with invalid patterns (snagel)
|
|
|
34
|
|
|
35 * NUTCH-1341 NotModified time set to now but page not modified (markus)
|
|
|
36
|
|
|
37 * NUTCH-1215 UpdateDB should not require segment as input (markus)
|
|
|
38
|
|
|
39 * NUTCH-1383 IndexingFiltersChecker to show error message instead of null pointer exception (snagel)
|
|
|
40
|
|
|
41 * NUTCH-1476 SegmentReader getStats should set parsed = -1 if no parsing took place (snagel)
|
|
|
42
|
|
|
43 * NUTCH-1252 SegmentReader -get shows wrong data (snagel)
|
|
|
44
|
|
|
45 * NUTCH-1344 BasicURLNormalizer to normalize https same as http (snagel)
|
|
|
46
|
|
|
47 * NUTCH-706 Url regex normalizer: pattern for session id removal not to match "newsId" (Meghna Kukreja via snagel)
|
|
|
48
|
|
|
49 * NUTCH-1415 release packages to contain top level folder apache-nutch-x.x (snagel)
|
|
|
50
|
|
|
51 * NUTCH-1441 AnchorIndexingFilter should use plain HashSet (ferdy via lewismc)
|
|
|
52
|
|
|
53 * NUTCH-1470 Ensure test files are included for runtime testing (lewismc)
|
|
|
54
|
|
|
55 * NUTCH-1434 Indexer to delete robots noindex (markus)
|
|
|
56
|
|
|
57 * NUTCH-1443 Solr schema version is invalid (markus)
|
|
|
58
|
|
|
59 * NUTCH-1417 Remove o.a.n.metadata.Office (lewismc)
|
|
|
60
|
|
|
61 * NUTCH-1376 Add description parameter to every ant task (lewismc)
|
|
|
62
|
|
|
63 * NUTCH-1440 reconfigure non-existent stopwords_en.txt in schema-solr4.xml (shekhar sharma via lewismc)
|
|
|
64
|
|
|
65 * NUTCH-1439 Define boost field as type float in schema-solr4.xml (shekhar sharma via lewismc)
|
|
|
66
|
|
|
67 * NUTCH-1433 Upgrade to Tika 1.2 (jnioche)
|
|
|
68
|
|
|
69 * NUTCH-1388 Optionally maintain custom fetch interval despite AdaptiveFetchSchedule (markus)
|
|
|
70
|
|
|
71 * NUTCH-1430 Freegenerator records overwrite CrawlDB records with AdaptiveFetchSchedule (markus)
|
|
|
72
|
|
|
73 * NUTCH-1087 Deprecate crawl command and replace with example script (jnioche)
|
|
|
74
|
|
|
75 * NUTCH-1306 Add option to not commit and clarify existing solr.commit.size (ferdy)
|
|
|
76
|
|
|
77 * NUTCH-1405 Allow to overwrite CrawlDatum's with injected entries (markus)
|
|
|
78
|
|
|
79 * NUTCH-1412 Upgrade commons lang (markus)
|
|
|
80
|
|
|
81 * NUTCH-1251 SolrDedup to use proper Lucene catch-all query (Arkadi Kosmynin via markus)
|
|
|
82
|
|
|
83 * NUTCH-1407 BasicIndexingFilter to optionally add domain field (markus)
|
|
|
84
|
|
|
85 * NUTCH-1408 RobotRulesParser main doesn't take URL's (markus)
|
|
|
86
|
|
|
87 * NUTCH-1300 Indexer to filter normalize URL's (markus)
|
|
|
88
|
|
|
89 * NUTCH-1330 WebGraph OutlinkDB to preserve back up (markus)
|
|
|
90
|
|
|
91 * NUTCH-1319 HostNormalizer plugin (markus)
|
|
|
92
|
|
|
93 * NUTCH-1386 Headings filter not to add empty values (markus)
|
|
|
94
|
|
|
95 * NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling (ferdy via markus)
|
|
|
96
|
|
|
97 * NUTCH-1352 Improve regex urlfilters/normalizers synchronization (ferdy via markus)
|
|
|
98
|
|
|
99 * NUTCH-1024 Dynamically set fetchInterval by MIME-type (markus)
|
|
|
100
|
|
|
101 * NUTCH-1364 Add a counter in Generator for malformed urls (lewismc)
|
|
|
102
|
|
|
103 * NUTCH-1262 Map `duplicating` content-types to a single type (markus)
|
|
|
104
|
|
|
105 * NUTCH-1385 More robust plug-in order properties in nutch-site.xml (Andy Xue via markus)
|
|
|
106
|
|
|
107 * NUTCH-1336 Optionally not index db_notmodified pages (markus)
|
|
|
108
|
|
|
109 * NUTCH-1346 Follow outlinks to ignore external (markus)
|
|
|
110
|
|
|
111 * NUTCH-1320 IndexChecker and ParseChecker choke on IDN's (markus)
|
|
|
112
|
|
|
113 * NUTCH-1351 DomainStatistics to aggregate by TLD (markus)
|
|
|
114
|
|
|
115 * NUTCH-1381 Allow to override default subcollection field name (markus)
|
|
|
116
|
|
|
117 * NUTCH-XX Commit to add configuration for separation of ant distribution targets (lewismc + jnioche)
|
|
|
118
|
|
|
119 Release 1.5.1 - 07/10/2012
|
|
|
120
|
|
|
121 * NUTCH-1404 Nutch script fails to find job file in deploy mode (sidabatra, jnioche)
|
|
|
122
|
|
|
123 * NUTCH-1415 release packages to contain top level folder apache-nutch-x.x (snagel via lewismc)
|
|
|
124
|
|
|
125 * NUTCH-1400 Remove developer -core option for bin/nutch (jnioche)
|
|
|
126
|
|
|
127 * NUTCH-1384 Typo in ParseSegment's run-method (Matthias Agethle via markus)
|
|
|
128
|
|
|
129 * NUTCH-1398 Upgrade to Hadoop 1.0.3 (jnioche)
|
|
|
130
|
|
|
131 Release 1.5 - 04/15/2012
|
|
|
132
|
|
|
133 * NUTCH-1208 Don't include KEYS file in bin distribution (jnioche)
|
|
|
134
|
|
|
135 * NUTCH-1234 Upgrade to Tika 1.1 (jnioche, markus)
|
|
|
136
|
|
|
137 * NUTCH-809 Parse-metatags plugin (jnioche)
|
|
|
138
|
|
|
139 * NUTCH-1310 Nutch to send HTTP-accept header (markus)
|
|
|
140
|
|
|
141 * NUTCH-1305 Domain(blacklist)URLFilter to trim entries (markus)
|
|
|
142
|
|
|
143 * NUTCH-1307 Improve formatting of ant targets for clearer project help (lewismc)
|
|
|
144
|
|
|
145 * NUTCH-1299 LinkRank inverter to ignore records without Node (markus)
|
|
|
146
|
|
|
147 * NUTCH-1258 MoreIndexingFilter should be able to read Content-Type from both parse metadata and content metadata (jnioche, markus)
|
|
|
148
|
|
|
149 * NUTCH-1293 IndexingFiltersChecker to store detected content type in crawldatum metadata (markus)
|
|
|
150
|
|
|
151 * NUTCH-1291 Fetcher to stringify exception on // unexpected exception (markus)
|
|
|
152
|
|
|
153 * NUTCH-965 Skip parsing for truncated documents (alexis, lewismc, ferdy)
|
|
|
154
|
|
|
155 * NUTCH-1210 DomainBlacklistFilter (markus)
|
|
|
156
|
|
|
157 * NUTCH-1193 Incorrect url transform to lowercase: parameter solr (Eduardo dos Santos Leggiero via lewismc)
|
|
|
158
|
|
|
159 * NUTCH-1272 Wrong property name for index-static in nutch-default.xml (Daniel Baur via jnioche)
|
|
|
160
|
|
|
161 * NUTCH-1259 Store detected content-type in crawldatum metadata (jnioche, markus)
|
|
|
162
|
|
|
163 * NUTCH-1266 Subcollection to optionally write to configured fields (markus)
|
|
|
164
|
|
|
165 * NUTCH-1005 Parse headings plugin (markus)
|
|
|
166
|
|
|
167 * NUTCH-1264 Configurable indexing plugin index-metadata (jnioche)
|
|
|
168
|
|
|
169 * NUTCH-1242 Allow disabling of URL Filters in ParseSegment (Edward Drapkin via markus)
|
|
|
170
|
|
|
171 * NUTCH-1256 WebGraph to dump host + score (markus)
|
|
|
172
|
|
|
173 * NUTCH-1260 Fetcher should log fetching of redirects (Sebastian Nagel via markus)
|
|
|
174
|
|
|
175 * NUTCH-1255 Change ivy.xml of all plugins to remove "nutch.root" property (ferdy)
|
|
|
176
|
|
|
177 * NUTCH-1248 Generator to select on status (markus)
|
|
|
178
|
|
|
179 * NUTCH-1177 Generator to select on retry interval (markus)
|
|
|
180
|
|
|
181 * NUTCH-1246 Upgrade to Hadoop 1.0.0 (jnioche)
|
|
|
182
|
|
|
183 * NUTCH-1139 Indexer to delete gone documents (markus)
|
|
|
184
|
|
|
185 * NUTCH-1244 CrawlDBDumper to filter by regex (markus)
|
|
|
186
|
|
|
187 * NUTCH-1237 Improve javac arguements for more verbose ouput (lewismc)
|
|
|
188
|
|
|
189 * NUTCH-1236 Add link to site documentation to download older versions of Nutch (lewismc)
|
|
|
190
|
|
|
191 * NUTCH-1146 Prevent generation of _SUCCESS files in output (jnioche)
|
|
|
192
|
|
|
193 * NUTCH-1232 Remove site field from index-basic (markus)
|
|
|
194
|
|
|
195 * NUTCH-1239 Webgraph should remove deleted pages from segment input (markus)
|
|
|
196
|
|
|
197 * NUTCH-1238 Fetcher throughput threshold must start before feeder finished (markus)
|
|
|
198
|
|
|
199 * NUTCH-1138 remove LogUtil from trunk and nutch gora (lewismc)
|
|
|
200
|
|
|
201 * NUTCH-1231 Upgrade to Tika 1.0 (markus)
|
|
|
202
|
|
|
203 * NUTCH-1230 MimeType API deprecated and breaks with Tika 1.0 (markus)
|
|
|
204
|
|
|
205 * NUTCH-1235 Upgrade to new Hadoop 0.20.205.0 (markus)
|
|
|
206
|
|
|
207 * NUTCH-1217 Update NOTICE.txt to drop some copyrights (lewismc)
|
|
|
208
|
|
|
209 * NUTCH-1129 Add freegenerator, domainstats and crawldbscanner to log4j (markus)
|
|
|
210
|
|
|
211 * NUTCH-1184 Fetcher to parse and follow Nth degree outlinks (markus)
|
|
|
212
|
|
|
213 * NUTCH-1221 Migrate DomainStatistics to MapReduce API (markus)
|
|
|
214
|
|
|
215 * NUTCH-1216 Add trivial comment to lib/native/README.txt (lewismc)
|
|
|
216
|
|
|
217 * NUTCH-1214 DomainStats tool should be named for what it's doing (markus)
|
|
|
218
|
|
|
219 * NUTCH-1213 Pass additional SolrParams when indexing to Solr (ab)
|
|
|
220
|
|
|
221 * NUTCH-1211 URLFilterChecker command line help doesn't inform user of
|
|
|
222 STDIN requirements (mattmann)
|
|
|
223
|
|
|
224 * NUTCH-1209 Output from ParserChecker Url missing a newline (mattmann)
|
|
|
225
|
|
|
226 * NUTCH-1207 ParserChecker to output signature (markus)
|
|
|
227
|
|
|
228 * NUTCH-1090 InvertLinks should inform when ignoring internal links (Marek Backmann via markus)
|
|
|
229
|
|
|
230 * NUTCH-1174 Outlinks are not properly normalized (markus)
|
|
|
231
|
|
|
232 * NUTCH-1203 ParseSegment to show number of milliseconds per parse (markus)
|
|
|
233
|
|
|
234 * NUTCH-1185 Decrease solr.commit.size to 250 (markus)
|
|
|
235
|
|
|
236 * NUTCH-1180 UpdateDB to backup previous CrawlDB (markus)
|
|
|
237
|
|
|
238 * NUTCH-1173 DomainStats doesn't count db_not_modified (markus)
|
|
|
239
|
|
|
240 * NUTCH-1155 Host/domain limit in generator is generate.max.count+1 (markus)
|
|
|
241
|
|
|
242 * NUTCH-1061 Migrate MoreIndexingFilter from Apache ORO to java.util.regex (markus)
|
|
|
243
|
|
|
244 * NUTCH-1178 Incorrect CSV header CrawlDatumCsvOutputFormat (markus)
|
|
|
245
|
|
|
246 * NUTCH-1142 Normalization and filtering in WebGraph (markus)
|
|
|
247
|
|
|
248 * NUTCH-1153 LinkRank not to log all keys and not to write Hadoop _SUCCESS file (markus)
|
|
|
249
|
|
|
250 Release 1.4 - 11/4/2011
|
|
|
251
|
|
|
252 * NUTCH-1195 Add Solr 4x (trunk) example schema (ab)
|
|
|
253
|
|
|
254 * NUTCH-1192 Add '/runtime' to svn ignore (ferdy)
|
|
|
255
|
|
|
256 * NUTCH-1097 application/xhtml+xml should be enabled in plugin.xml of parse-html; allow multiple mimetypes for plugin.xml (Ferdy via lewismc)
|
|
|
257
|
|
|
258 * NUTCH-797 Fix parse-tika and parse-html to use relative URL resolution per RFC-3986
|
|
|
259 (Robert Hohman, ab)
|
|
|
260
|
|
|
261 * NUTCH-1154 Upgrade to Tika 0.10. NOTE: Tika's new RTF parser may ignore more
|
|
|
262 text in malformed documents than previously - see TIKA-748 for details. (ab)
|
|
|
263
|
|
|
264 * NUTCH-1109 Add Sonar targets to Ant build.xml (lewismc)
|
|
|
265
|
|
|
266 * NUTCH-1152 Upgrade SolrJ to version 3.4.0 (ab)
|
|
|
267
|
|
|
268 * NUTCH-1136 Ant pmd target is broken (lewismc)
|
|
|
269
|
|
|
270 * NUTCH-1058 Upgrade Solr schema to version 1.4 (markus)
|
|
|
271
|
|
|
272 * NUTCH-1137 LinkDB invertlinks other options ignored when using -dir option (Sebastian Nagel, markus)
|
|
|
273
|
|
|
274 * NUTCH-1141 Configurable Fetcher queue depth (jnioche)
|
|
|
275
|
|
|
276 * NUTCH-1091 Remove commons logging dependency from Nutch branch and trunk (lewismc)
|
|
|
277
|
|
|
278 * NUTCH-672 allow unit tests to be run from bin/nutch (Todd Lipton via lewismc)
|
|
|
279
|
|
|
280 * NUTCH-937 Put plugins in classes/plugins in job file (Claudio Martella, Ferdy Galema, jnioche)
|
|
|
281
|
|
|
282 * NUTCH-623 Change plugin source directory "languageidentifier" to "language-identifier" (lewismc)
|
|
|
283
|
|
|
284 * NUTCH-1074 topN is ignored with maxNumSegments and generate.max.count (Robert Thomson via markus)
|
|
|
285
|
|
|
286 * NUTCH-1078 Upgrade all instances of commons logging to slf4j (with log4j backend) (lewismc)
|
|
|
287
|
|
|
288 * NUTCH-1115 Option to disable fixing embedded URL parameters in DomContentUtils (markus)
|
|
|
289
|
|
|
290 * NUTCH-1114 Attr file missing in domain filter (markus)
|
|
|
291
|
|
|
292 * NUTCH-1067 Configure minimum throughput for fetcher (markus)
|
|
|
293
|
|
|
294 * NUTCH-1102 Fetcher to rely on fetcher.parse directive (markus)
|
|
|
295
|
|
|
296 * NUTCH-1110 UpdateDB must not write _success file (markus)
|
|
|
297
|
|
|
298 * NUTCH-1105 Max content length option for index-basic (markus)
|
|
|
299
|
|
|
300 * NUTCH-940 static field plugin (Claudio Martella via lewismc)
|
|
|
301
|
|
|
302 * NUTCH-914 Implement Apache Project Branding Requirements (lewismc)
|
|
|
303
|
|
|
304 * NUTCH-1095 remove i18n from Nutch site to archive and legacy secton of wiki (lewismc)
|
|
|
305
|
|
|
306 * NUTCH-1101 Option to purge db_gone records with updatedb (markus)
|
|
|
307
|
|
|
308 * NUTCH-1096 Empty (not null) ContentLength results in failure of fetch (Ferdy Galema via jnioche)
|
|
|
309
|
|
|
310 * NUTCH-1073 Rename parameters 'fetcher.threads.per.host.by.ip' and 'fetcher.threads.per.host' (jnioche)
|
|
|
311
|
|
|
312 * NUTCH-1089 Short compressed pages caused exception in protocol-httpclient (Simone Frenzel via jnioche)
|
|
|
313
|
|
|
314 * NUTCH-1085 Nutch script does not require HADOOP_HOME (jnioche)
|
|
|
315
|
|
|
316 * NUTCH-1075 Delegate language identification to Tika (jnioche)
|
|
|
317
|
|
|
318 * NUTCH-1049 Add classes to bin/nutch script (markus)
|
|
|
319
|
|
|
320 * NUTCH-1051 Export WebGraph node scores for Solr.ExternalFileField (markus)
|
|
|
321
|
|
|
322 * NUTCH-1083 ParserChecker implements Tools (jnioche)
|
|
|
323
|
|
|
324 * NUTCH-1082 IndexingFiltersChecker utility does not list multi valued fields (markus)
|
|
|
325
|
|
|
326 * NUTCH-1004 Do not index empty values for title field (markus)
|
|
|
327
|
|
|
328 * NUTCH-914 Implement Apache Project Branding Requirements (lewismc via jnioche)
|
|
|
329
|
|
|
330 * NUTCH-1069 Readlinkdb broken on Hadoop > 0.20 (markus)
|
|
|
331
|
|
|
332 * NUTCH-1044 Redirected URLs and possibly all of their outlinked URLs have invalid scores (jnioche)
|
|
|
333
|
|
|
334 * NUTCH-1028 Log urls when parsing (markus)
|
|
|
335
|
|
|
336 * NUTCH-1065 New mvn.template (lewismc)
|
|
|
337
|
|
|
338 * NUTCH-1072 Display number and size of queues in Fetcher status (jnioche)
|
|
|
339
|
|
|
340 * NUTCH-1071 Crawldb update displays total number of URLs per status (jnioche)
|
|
|
341
|
|
|
342 * NUTCH-1045 MimeUtil to rely on default config provided by Tika (jnioche)
|
|
|
343
|
|
|
344 * NUTCH-1057 Fetcher thread time out configurable (markus)
|
|
|
345
|
|
|
346 * NUTCH-1037 Option to deduplicate anchors prior to indexing (markus)
|
|
|
347
|
|
|
348 * NUTCH-1050 Add segmentDir option to WebGraph (markus)
|
|
|
349
|
|
|
350 * NUTCH-1055 upgrade package.html file in language identifier plugin (lewismc)
|
|
|
351
|
|
|
352 * NUTCH-1059 Remove convdb command from /bin/nutch (lewismc)
|
|
|
353
|
|
|
354 * NUTCH-1019 Edit comment in org.apache.nutch.crawl.Crawl to reflect removal of legacy (lewismc)
|
|
|
355
|
|
|
356 * NUTCH-1023 Trivial error in error message for org.apache.nutch.crawl.LinkDbReader (lewismc)
|
|
|
357
|
|
|
358 * NUTCH-1043 Add pattern for filtering .js in default url filters (jnioche)
|
|
|
359
|
|
|
360 * NUTCH-1054 LinkDB optional during indexing (jnioche)
|
|
|
361
|
|
|
362 * NUTCH-1029 Readdb throws EOFException (markus)
|
|
|
363
|
|
|
364 * NUTCH-1036 Solr jobs should increment counters in Reporter (markus)
|
|
|
365
|
|
|
366 * NUTCH-987 Support HTTP auth for Solr communication (markus)
|
|
|
367
|
|
|
368 * NUTCH-1027 Degrade log level of `can't find rules for scope` (markus)
|
|
|
369
|
|
|
370 * NUTCH-783 IndexingFiltersChecker utility (jnioche via markus)
|
|
|
371
|
|
|
372 * NUTCH-1030 WebgraphDB program requires manually added directories (markus)
|
|
|
373
|
|
|
374 * NUTCH-1011 Normalize duplicate slashes in URL's (markus)
|
|
|
375
|
|
|
376 * NUTCH-993 NullPointerException at FetcherOutputFormat.checkOutputSpecs (Christian Guegi via jnioche)
|
|
|
377
|
|
|
378 * NUTCH-1013 Migrate RegexURLNormalizer from Apache ORO to java.util.regex (markus)
|
|
|
379
|
|
|
380 * NUTCH-1016 Strip UTF-8 non-character codepoints and add logging for SolrWriter (markus)
|
|
|
381
|
|
|
382 * NUTCH-1012 Cannot handle illegal charset $charset (markus)
|
|
|
383
|
|
|
384 * NUTCH-1022 Upgrade version number of Nutch agent in conf (markus)
|
|
|
385
|
|
|
386 * NUTCH-295 Description for fetcher.threads.fetch property (kubes via markus)
|
|
|
387
|
|
|
388 * NUTCH-1000 Add option not to commit to Solr (markus)
|
|
|
389
|
|
|
390 * NUTCH-1006 MetaEquiv with single quotes not accepted (markus)
|
|
|
391
|
|
|
392 * NUTCH-1010 ContentLength not trimmed (markus)
|
|
|
393
|
|
|
394 Release 1.3 - 6/4/2011
|
|
|
395
|
|
|
396 * NUTCH-995 Generate POM file using the Ivy makepom task (mattmann, jnioche, Gabriele Kahlout)
|
|
|
397
|
|
|
398 * NUTCH-1003 task 'package' does not reflect the new organisation of the code (jnioche)
|
|
|
399
|
|
|
400 * NUTCH-994 Fine tune Solr schema (markus)
|
|
|
401
|
|
|
402 * NUTCH-997 IndexingFitlers to store Date objects instead of Strings (jnioche)
|
|
|
403
|
|
|
404 * NUTCH-996 Indexer adds solr.commit.size+1 docs (markus)
|
|
|
405
|
|
|
406 * NUTCH-983 Upgrade SolrJ to 3.1 (markus, jnioche)
|
|
|
407
|
|
|
408 * NUTCH-989 Index-basic plugin and Solr schema now use date fieldType for tstamp field (markus)
|
|
|
409
|
|
|
410 * NUTCH-888 Remove parse-rss and add tests for rss to parse-tika (jnioche)
|
|
|
411
|
|
|
412 * NUTCH-991 SolrDedup must issue a commit (markus)
|
|
|
413
|
|
|
414 * NUTCH 986 SolrDedup fails due to date incorrect format (markus)
|
|
|
415
|
|
|
416 * NUTCH-977 SolrMappingReader uses hardcoded configuration parameter name for mapping file (markus)
|
|
|
417
|
|
|
418 * NUTCH-976 Rename properties solrindex.* to solr.* (markus)
|
|
|
419
|
|
|
420 * NUTCH-890 Fix IllegalAccessError with slf4j used in Solrj (markus)
|
|
|
421
|
|
|
422 * NUTCH-891 Subcollection plugin won't require blacklist any more (markus)
|
|
|
423
|
|
|
424 * NUTCH-972 CrawlDbMerger doesn't break on non-existent input (Gabriele Kahlout via jnioche)
|
|
|
425
|
|
|
426 * NUTCH-967 Upgrade to Tika 0.9 (jnioche)
|
|
|
427
|
|
|
428 * NUTCH-975 Fix missing/wrong headers in source files (markus)
|
|
|
429
|
|
|
430 * NUTCH-963 Add support for deleting Solr documents with STATUS_DB_GONE in CrawlDB (Claudio Martella, markus)
|
|
|
431
|
|
|
432 * NUTCH-825 Publish nutch artifacts to central maven repository (mattmann, jnioche)
|
|
|
433
|
|
|
434 * NUTCH-962 max. redirects not handled correctly: fetcher stops at max-1 redirects (Sebastian Nagel via ab)
|
|
|
435
|
|
|
436 * NUTCH-921 Reduce dependency of Nutch on config files (ab)
|
|
|
437
|
|
|
438 * NUTCH-876 Remove remaining robots/IP blocking code in lib-http (ab)
|
|
|
439
|
|
|
440 * NUTCH-872 Change the default fetcher.parse to FALSE (ab)
|
|
|
441
|
|
|
442 * NUTCH-564 External parser supports encoding attribute (Antony Bowesman, mattmann)
|
|
|
443
|
|
|
444 * NUTCH-964 Upgraded Xerces to 2.91, ERROR conf.Configuration - Failed to set setXIncludeAware (markus)
|
|
|
445
|
|
|
446 * NUTCH-927 Fetcher.timelimit.mins is invalid when depth is greater than 1 (Wade Lau via jnioche)
|
|
|
447
|
|
|
448 * NUTCH-824 Crawling - File Error 404 when fetching file with an hexadecimal character in the file name (Michela Becchi via jnioche)
|
|
|
449
|
|
|
450 * NUTCH-954 Strict application of Content-Length limit for http protocols (Alexis Detreglode via jnioche)
|
|
|
451
|
|
|
452 * NUTCH-950 DomainURLFilter throws NPE on bogus urls (Alexis Detreglode via jnioche)
|
|
|
453
|
|
|
454 * NUTCH-935 basicurlnormalizer removes unnecessary /./ in URLs (Stondet via markus)
|
|
|
455
|
|
|
456 * NUTCH-912 MoreIndexingFilter does not parse docx and xlsx date formats (Markus Jelsma, jnioche)
|
|
|
457
|
|
|
458 * NUTCH-886 A .gitignore file for Nutch (dogacan)
|
|
|
459
|
|
|
460 * NUTCH-930 Remove remaining dependencies on Lucene API (ab)
|
|
|
461
|
|
|
462 * NUTCH-883 Remove unused parameters from nutch-default.xml (jnioche)
|
|
|
463
|
|
|
464 * NUTCH-936 LanguageIdentifier should not set empty lang field on NutchDocument (Markus Jelsma via jnioche)
|
|
|
465
|
|
|
466 * NUTCH-787 ScoringFilters should not override the injected score (jnioche)
|
|
|
467
|
|
|
468 * NUTCH-949 Conflicting ANT jars in classpath (jnioche)
|
|
|
469
|
|
|
470 * NUTCH-863 Benchmark and a testbed proxy server (ab)
|
|
|
471
|
|
|
472 * NUTCH-844 Improve NutchConfiguration (ab)
|
|
|
473
|
|
|
474 * NUTCH-845 Native hadoop libs not available through maven (ab)
|
|
|
475
|
|
|
476 * NUTCH-843 Separate the build and runtime environments (ab)
|
|
|
477
|
|
|
478 * NUTCH-821 Use ivy in nutch builds (Enis Soztutar, jnioche)
|
|
|
479
|
|
|
480 * NUTCH-837 Remove search servers and Lucene dependencies (ab)
|
|
|
481
|
|
|
482 * NUTCH-836 Remove deprecated parse plugins (jnioche)
|
|
|
483
|
|
|
484 * NUTCH-939 Added -dir command line option to SolrIndexer (Claudio Martella via ab)
|
|
|
485
|
|
|
486 * NUTCH-948 Remove Lucene dependencies (ab)
|
|
|
487
|
|
|
488 Release 1.2 - 09/18/2010
|
|
|
489
|
|
|
490 * NUTCH-901 Make index-more plug-in configurable (Markus Jelsma via mattmann)
|
|
|
491
|
|
|
492 * NUTCH-908 Infinite Loop and Null Pointer Bugs in Searching (kubes via mattmann)
|
|
|
493
|
|
|
494 * NUTCH-906 Nutch OpenSearch sometimes raises DOMExceptions (Asheesh Laroia via ab)
|
|
|
495
|
|
|
496 * NUTCH-862 HttpClient null pointer exception (Sebastian Nagel via ab)
|
|
|
497
|
|
|
498 * NUTCH-905 Configurable file protocol parent directory crawling (Thorsten Scherler, mattmann, ab)
|
|
|
499
|
|
|
500 * NUTCH-877 Allow setting of slop values for non-quote phrase queries on query-basic plugin (kubes via jnioche)
|
|
|
501
|
|
|
502 * NUTCH-716 Make subcollection index filed multivalued (Dmitry Lihachev via jnioche)
|
|
|
503
|
|
|
504 * NUTCH-878 ScoringFilters should not override the injected score
|
|
|
505
|
|
|
506 * NUTCH-870 Injector should add the metadata before calling injectedScore (jnioche via mattmann)
|
|
|
507
|
|
|
508 * NUTCH-858 No longer able to set per-field boosts on lucene documents (ab)
|
|
|
509
|
|
|
510 * NUTCH-869 Add parse-html back (jnioche)
|
|
|
511
|
|
|
512 * NUTCH-871 MoreIndexingFilter missing date format (Max Lynch via mattmann)
|
|
|
513
|
|
|
514 * NUTCH-696 Timeout for Parser (ab, jnioche)
|
|
|
515
|
|
|
516 * NUTCH-857 DistributedBeans should not close their RPC counterparts (kubes)
|
|
|
517
|
|
|
518 * NUTCH-855 ScoringFilter and IndexingFilter: To allow for the propagation of URL Metatags
|
|
|
519 and their subsequent indexing (Scott Gonyea via mattmann)
|
|
|
520
|
|
|
521 * NUTCH-677 Segment merge filering based on segment content (Marcin Okraszewski via mattmann)
|
|
|
522
|
|
|
523 * NUTCH-774 Retry interval in crawl date is set to 0 (Reinhard Schwab via mattmann)
|
|
|
524
|
|
|
525 * NUTCH-697 Generate log output for solr indexer and dedup (Dmitry Lihachev, Jeroen van Vianen via mattmann)
|
|
|
526
|
|
|
527 * NUTCH-850 SolrDeleteDuplicates needs to clone the SolrRecord objects (jnioche)
|
|
|
528
|
|
|
529 * NUTCH-838 Add timing information to all Tool classes (Jeroen van Vianen, mattmann)
|
|
|
530
|
|
|
531 * NUTCH-835 Document deduplication failed using MD5Signature (Sebastian Nagel via ab)
|
|
|
532
|
|
|
533 * NUTCH-831 Allow configuration of how fields crawled by Nutch are stored / indexed /
|
|
|
534 tokenized (Jeroen van Vianen via mattmann)
|
|
|
535
|
|
|
536 * NUTCH-278 Fetcher-status might need clarification: kbit/s instead of kb/s shown (Alex McLintock via mattmann)
|
|
|
537
|
|
|
538 * NUTCH-833 Website is still Lucene branded (mattmann, Alex McLintock)
|
|
|
539
|
|
|
540 * NUTCH-832 Website menu has lots of broken links - in particular the API docs (Alex McLintock via mattmann)
|
|
|
541
|
|
|
542 Release 1.1 - 2010-06-06
|
|
|
543
|
|
|
544 * NUTCH-819 Included Solr schema.xml and solrindex-mapping.xml don't play together (ab)
|
|
|
545
|
|
|
546 * NUTCH-818 Bugfix : Parse-tika uses minorCodes instead of majorCodes in ParseStatus (jnioche)
|
|
|
547
|
|
|
548 * NUTCH-816 Add zip target to build.xml (mattmann)
|
|
|
549
|
|
|
550 * NUTCH-732 Subcollection plugin not working (Filipe Antunes, ab)
|
|
|
551
|
|
|
552 * NUTCH-815 Invalid blank line before If-Modified-Since header (Pascal Dimassimo via ab)
|
|
|
553
|
|
|
554 * NUTCH-814 SegmentMerger bug (Rob Bradshaw, ab)
|
|
|
555
|
|
|
556 * NUTCH-812 Crawl.java incorrectly uses the Generator API resulting in NPE (Phil Barnett via mattmann and ab)
|
|
|
557
|
|
|
558 * NUTCH-810 Upgrade to Tika 0.7 (jnioche)
|
|
|
559
|
|
|
560 * NUTCH-785 Copy metadata from origin URL when redirecting in Fetcher + call scfilters.initialScore on newly created URL (jnioche)
|
|
|
561
|
|
|
562 * NUTCH-779 Mechanism for passing metadata from parse to crawldb (jnioche)
|
|
|
563
|
|
|
564 * NUTCH-784 CrawlDBScanner (jnioche)
|
|
|
565
|
|
|
566 * NUTCH-762 Generator can generate several segments in one parse of the crawlDB (jnioche)
|
|
|
567
|
|
|
568 * NUTCH-740 Configuration option to override default language for fetched pages (Marcin Okraszewski via jnioche)
|
|
|
569
|
|
|
570 * NUTCH-803 Upgrade to Hadoop 0.20.2 (ab)
|
|
|
571
|
|
|
572 * NUTCH-787 Upgrade Lucene to 3.0.1. (Dawid Weiss via ab)
|
|
|
573
|
|
|
574 * NUTCH-796 Zero results problems difficult to troubleshoot due to lack of logging (ab)
|
|
|
575
|
|
|
576 * NUTCH-801 Remove RTF and MP3 parse plugins (jnioche)
|
|
|
577
|
|
|
578 * NUTCH-798 Upgrade to SOLR1.4 and its dependencies (jnioche)
|
|
|
579
|
|
|
580 * NUTCH-799 SOLRIndexer to commit once all reducers have finished (jnioche)
|
|
|
581
|
|
|
582 * NUTCH-782 Ability to order htmlparsefilters (jnioche)
|
|
|
583
|
|
|
584 * NUTCH-719 fetchQueues.totalSize incorrect in Fetcher (Steven Denny via jnioche)
|
|
|
585
|
|
|
586 * NUTCH-790 Some external javadoc links are broken (siren)
|
|
|
587
|
|
|
588 * NUTCH-766 Tika parser (jnioche via mattmann)
|
|
|
589
|
|
|
590 * NUTCH-786 Improvement to the list of suffix domains (jnioche)
|
|
|
591
|
|
|
592 * NUTCH-775 Enhance searcher interface (siren)
|
|
|
593
|
|
|
594 * NUTCH-781 Update Tika to v0.6 (jnioche)
|
|
|
595
|
|
|
596 * NUTCH-269 CrawlDbReducer: OOME because no upper-bound on inlinks count (stack + jnioche)
|
|
|
597
|
|
|
598 * NUTCH-655 Injecting Crawl metadata (jnioche)
|
|
|
599
|
|
|
600 * NUTCH-658 Use counters to report fetching and parsing status (jnioche)
|
|
|
601
|
|
|
602 * NUTCH-777 Upgrading to jetty6 broke unit tests (mattmann)
|
|
|
603
|
|
|
604 * NUTCH-767 Update Tika to v0.5 for the MimeType detection (Julien Nioche via ab)
|
|
|
605
|
|
|
606 * NUTCH-769 Fetcher to skip queues for URLS getting repeated exceptions
|
|
|
607 (Julien Nioche via ab)
|
|
|
608
|
|
|
609 * NUTCH-768 - Upgrade Nutch 1.0 to use Hadoop 0.20.1, also upgrades Xerces to
|
|
|
610 version 2.9.1. (kubes)
|
|
|
611
|
|
|
612 * NUTCH-712 ParseOutputFormat should catch java.net.MalformedURLException
|
|
|
613 coming from normalizers (Julien Nioche via ab)
|
|
|
614
|
|
|
615 * NUTCH-741 Job file includes multiple copies of nutch config files
|
|
|
616 (Kirby Bohling via ab)
|
|
|
617
|
|
|
618 * NUTCH-739 SolrDeleteDuplications too slow when using hadoop (Dmitry Lihachev via ab)
|
|
|
619
|
|
|
620 * NUTCH-738 Close SegmentUpdater when FetchedSegments is closed
|
|
|
621 (Martina Koch, Kirby Bohling via ab)
|
|
|
622
|
|
|
623 * NUTCH-746 NutchBeanConstructor does not close NutchBean upon contextDestroyed,
|
|
|
624 causing resource leak in the container. (Kirby Bohling via ab)
|
|
|
625
|
|
|
626 * NUTCH-772 Upgrade Nutch to use Lucene 2.9.1 (ab)
|
|
|
627
|
|
|
628 * NUTCH-760 Allow field mapping from Nutch to Solr index (David Stuart, ab)
|
|
|
629
|
|
|
630 * NUTCH-761 Avoid cloning CrawlDatum in CrawlDbReducer (Julien Nioche, ab)
|
|
|
631
|
|
|
632 * NUTCH-753 Prevent new Fetcher from retrieving the robots twice (Julien Nioche via ab)
|
|
|
633
|
|
|
634 * NUTCH-773 - Some minor bugs in AbstractFetchSchedule (Reinhard Schwab via ab)
|
|
|
635
|
|
|
636 * NUTCH-765 - Allow Crawl class to call Either Solr or Lucene Indexer (kubes)
|
|
|
637
|
|
|
638 * NUTCH-735 - crawl-tool.xml must be read before nutch-site.xml when
|
|
|
639 invoked using crawl command (Susam Pal via dogacan)
|
|
|
640
|
|
|
641 * NUTCH-721 - Fetcher2 Slow (Julien Nioche via dogacan)
|
|
|
642
|
|
|
643 * NUTCH-702 - Lazy Instanciation of Metadata in CrawlDatum (Julien Nioche via dogacan)
|
|
|
644
|
|
|
645 * NUTCH-707 - Generation of multiple segments in multiple runs returns only 1 segment
|
|
|
646 (Michael Chen, ab)
|
|
|
647
|
|
|
648 * NUTCH-730 - NPE in LinkRank if no nodes with which to create the WebGraph
|
|
|
649 (Dennis Kubes via ab)
|
|
|
650
|
|
|
651 * NUTCH-731 - Redirection of robots.txt in RobotRulesParser (Julien Nioche via ab)
|
|
|
652
|
|
|
653 * NUTCH-757 - RequestUtils getBooleanParameter() always returns false
|
|
|
654 (Niall Pemberton via ab)
|
|
|
655
|
|
|
656 * NUTCH-754 - Use GenericOptionsParser instead of FileSystem.parseArgs() (Julien
|
|
|
657 Nioche via ab)
|
|
|
658
|
|
|
659 * NUTCH-756 - CrawlDatum.set() does not reset Metadata if it is null (Julien Nioche
|
|
|
660 via ab)
|
|
|
661
|
|
|
662 * NUTCH-679 - Fetcher2 implementing Tool (Julien Nioche via ab)
|
|
|
663
|
|
|
664 * NUTCH-758 - Set subversion eol-style to "native" (Niall Pemberton via ab)
|
|
|
665
|
|
|
666 Release 1.0 - 2009-03-23
|
|
|
667
|
|
|
668 1. NUTCH-474 - Fetcher2 crawlDelay and blocking fix (Dogacan Guney via ab)
|
|
|
669
|
|
|
670 2. NUTCH-443 - Allow parsers to return multiple Parse objects.
|
|
|
671 (Dogacan Guney et al, via ab)
|
|
|
672
|
|
|
673 3. NUTCH-393 - Indexer should handle null documents returned by filters.
|
|
|
674 (Eelco Lempsink via ab)
|
|
|
675
|
|
|
676 4. NUTCH-456 - Parse msexcel plugin speedup (Heiko Dietze via siren)
|
|
|
677
|
|
|
678 5. NUTCH-446 - RobotRulesParser should ignore Crawl-delay values of other
|
|
|
679 bots in robots.txt (Dogacan Guney via siren)
|
|
|
680
|
|
|
681 6. NUTCH-482 - Remove redundant plugin lib-log4j (siren)
|
|
|
682
|
|
|
683 7. NUTCH-483 - Remove redundant commons-logging jar from ontology plugin
|
|
|
684 (siren)
|
|
|
685
|
|
|
686 8. NUTCH-161 - Change Plain text parser to
|
|
|
687 use parser.character.encoding.default property for fall back encoding
|
|
|
688 (KuroSaka TeruHiko, siren)
|
|
|
689
|
|
|
690 9. NUTCH-61 - Support for adaptive re-fetch interval and detection of
|
|
|
691 unmodified content. (ab)
|
|
|
692
|
|
|
693 10. NUTCH-392 - OutputFormat implementations should pass on Progressable.
|
|
|
694 (cutting via ab)
|
|
|
695
|
|
|
696 11. NUTCH-495 - Unnecessary delays in Fetcher2 (dogacan)
|
|
|
697
|
|
|
698 12. NUTCH-443 - allow parsers to return multiple Parse object, this will speed
|
|
|
699 up the rss parser (dogacan via mattmann). This update is a fix and semantics
|
|
|
700 change from the original patch for NUTCH-443. The original patch did not tell
|
|
|
701 the Indexer to read crawl_parse too so that it can pickup sub-urls' fetch
|
|
|
702 datums. This patch addresses that issue. Now, if Fetcher gets a null content,
|
|
|
703 instead of pushing an empty content, it filters the null content.
|
|
|
704
|
|
|
705 13. NUTCH-485 - Change HtmlParseFilter 's to return ParseResult object instead of
|
|
|
706 Parse object. (Gal Nitzan via dogacan)
|
|
|
707
|
|
|
708 14. NUTCH-489 - URLFilter-suffix management of the url path when the url contains
|
|
|
709 some query parameters. (Emmanuel Joke via dogacan)
|
|
|
710
|
|
|
711 15. NUTCH-502 - Bug in SegmentReader causes infinite loop.
|
|
|
712 (Ilya Vishnevsky via dogacan)
|
|
|
713
|
|
|
714 16. NUTCH-444 Possibly use a different library to parse RSS feed for improved
|
|
|
715 performance and compatibility. This patch introduced a new plugin, feed,
|
|
|
716 that includes an index filter and a parse plugin for feeds that uses ROME.
|
|
|
717 There was discussion to remove parse-rss, in light of the feed plugin,
|
|
|
718 however, this patch does not explicitly remove parse-rss. (dogacan, mattmann)
|
|
|
719
|
|
|
720 17. NUTCH-471 - Fix synchronization in NutchBean creation.
|
|
|
721 (Enis Soztutar via dogacan)
|
|
|
722
|
|
|
723 18. Upgrade to Lucene 2.2.0 and Hadoop 0.12.3. (ab)
|
|
|
724
|
|
|
725 19. NUTCH-468 - Scoring filter should distribute score to all outlinks at
|
|
|
726 once. (dogacan)
|
|
|
727
|
|
|
728 20. NUTCH-504 - NUTCH-443 broke parsing during fetching. (dogacan)
|
|
|
729
|
|
|
730 21. NUTCH-497 - Extreme Nested Tags causes StackOverflowException in
|
|
|
731 DomContentUtils...Spider Trap. (kubes)
|
|
|
732
|
|
|
733 22. NUTCH-434 - Replace usage of ObjectWritable with something based on
|
|
|
734 GenericWritable. (dogacan)
|
|
|
735
|
|
|
736 23. NUTCH-499 - Refactor LinkDb and LinkDbMerger to reuse code. (dogacan)
|
|
|
737
|
|
|
738 24. NUTCH-498 - Use Combiner in LinkDb to increase speed of linkdb generation.
|
|
|
739 (Espen Amble Kolstad via dogacan)
|
|
|
740
|
|
|
741 25. NUTCH-507 - lib-lucene-analyzers jar defintion is wrong in plugin.xml.
|
|
|
742 (Emmanuel Joke via dogacan)
|
|
|
743
|
|
|
744 26. NUTCH-503 - Generator exits incorrectly for small fetchlists.
|
|
|
745 (Vishal Shah via dogacan)
|
|
|
746
|
|
|
747 27. NUTCH-505 - Outlink urls should be validated. (dogacan)
|
|
|
748
|
|
|
749 28. NUTCH-510 - IndexMerger delete working dir. (Enis Soztutar via dogacan)
|
|
|
750
|
|
|
751 29. NUTCH-513 - suffix-urlfilter.txt does not have a template. (dogacan)
|
|
|
752
|
|
|
753 30. NUTCH-515 - Next fetch time is set incorrectly. (dogacan)
|
|
|
754
|
|
|
755 30. NUTCH-506 - Nutch should delegate compression to Hadoop. (dogacan)
|
|
|
756
|
|
|
757 31. NUTCH-517 - build encoding should be UTF-8. (Enis Soztutar via dogacan).
|
|
|
758
|
|
|
759 32. NUTCH-518 - Fix OpicScoringFilter to respect scoring filter chaining.
|
|
|
760 (Enis Soztutar via dogacan)
|
|
|
761
|
|
|
762 33. NUTCH-516 - Next fetch time is not set when it is a
|
|
|
763 CrawlDatum.STATUS_FETCH_GONE. (Emmanuel Joke via dogacan)
|
|
|
764
|
|
|
765 34. NUTCH-525 - DeleteDuplicates generates ArrayIndexOutOfBoundsException
|
|
|
766 when trying to rerun dedup on a segment. (Vishal Shah via dogacan)
|
|
|
767
|
|
|
768 35. NUTCH-514 - Indexer should only index pages with fetch status SUCCESS.
|
|
|
769 (dogacan) Note: There is a bigger problem, i.e how to deal
|
|
|
770 with redirected pages, and this issue can be considered as a band-aid
|
|
|
771 for the time being. See NUTCH-273 and NUTCH-353 for more details.
|
|
|
772
|
|
|
773 36. NUTCH-533 - LinkDbMerger: url normalized is not updated in the key and
|
|
|
774 inlinks list. (Emmanuel Joke via dogacan)
|
|
|
775
|
|
|
776 37. NUTCH-535 -ParseData's contentMeta accumulates unnecessary values during
|
|
|
777 parse. (dogacan)
|
|
|
778
|
|
|
779 38. NUTCH-522 - Use URLValidator in the Injector. (Emmanuel Joke, dogacan)
|
|
|
780
|
|
|
781 39. NUTCH-536 - Reduce number of warnings in nutch core. (dogacan)
|
|
|
782
|
|
|
783 40. NUTCH-439 - Top Level Domains Indexing / Scoring. Also adds
|
|
|
784 domain-related utilities. (Enis Soztutar via dogacan)
|
|
|
785
|
|
|
786 41. NUTCH-544 - Upgrade Carrot2 clustering plugin to the newest stable
|
|
|
787 release (2.1). (Dawid Weiss via dogacan)
|
|
|
788
|
|
|
789 42. NUTCH-545 - Configuration and OnlineClusterer get initialized in every
|
|
|
790 request. (Dawid Weiss via dogacan)
|
|
|
791
|
|
|
792 43. NUTCH-532 - CrawlDbMerger: wrong computation of last fetch time.
|
|
|
793 (Emmanuel Joke via dogacan)
|
|
|
794
|
|
|
795 44. NUTCH-550 - Parse fails if db.max.outlinks.per.page is -1. (dogacan)
|
|
|
796
|
|
|
797 45. NUTCH-546 - file URL are filtered out by the crawler. (dogacan)
|
|
|
798
|
|
|
799 46. NUTCH-554 - Generator throws IOException on invalid urls.
|
|
|
800 (Brian Whitman via ab)
|
|
|
801
|
|
|
802 47. NUTCH-529 - NodeWalker.skipChildren doesn't work for more than 1 child.
|
|
|
803 (Emmanuel Joke via dogacan)
|
|
|
804
|
|
|
805 48. NUTCH-25 - needs 'character encoding' detector.
|
|
|
806 (Doug Cook, dogacan, Marcin Okraszewski, Renaud Richardet via dogacan)
|
|
|
807
|
|
|
808 49. NUTCH-508 - ${hadoop.log.dir} and ${hadoop.log.file} are not propagated
|
|
|
809 to the tasktracker. (Mathijs Homminga, Emmanuel Joke via dogacan)
|
|
|
810
|
|
|
811 50. NUTCH-562 - Port mime type framework to use Tika mime detection framework.
|
|
|
812 (mattmann)
|
|
|
813
|
|
|
814 51. NUTCH-488 - Avoid parsing uneccessary links and get a more relevant outlink
|
|
|
815 list. (Emmanuel Joke, Marcin Okraszewski via kubes)
|
|
|
816
|
|
|
817 52. NUTCH-501 - Implement a different caching mechanism for objects cached in
|
|
|
818 configuration. (dogacan)
|
|
|
819
|
|
|
820 53. NUTCH-552 - Upgrade Nutch to Hadoop 0.15.x. (kubes)
|
|
|
821
|
|
|
822 54. NUTCH-565 - Arc File to Nutch Segments Converter. (kubes)
|
|
|
823
|
|
|
824 55. NUTCH-547 - Redirection handling: YahooSlurp's algorithm.
|
|
|
825 (dogacan, kubes via dogacan)
|
|
|
826
|
|
|
827 56. NUTCH-548 - Move URLNormalizer from Outlink to ParseOutputFormat.
|
|
|
828 (Emmanuel Joke via dogacan)
|
|
|
829
|
|
|
830 57. NUTCH-538 - Delete unused classes under o.a.n.util. (dogacan)
|
|
|
831
|
|
|
832 58. NUTCH-494 - FindBugs: CrawlDbReader and DeleteDuplicates. (dogacan)
|
|
|
833
|
|
|
834 59. NUTCH-574 - Including inlink anchor text in index can create irrelevant
|
|
|
835 search results. Created index-anchor plugin, removed functionality from
|
|
|
836 index-basic plugin. For backwards compatibility, add index-anchor plugin to
|
|
|
837 nutch-site.xml plugin.includes. (kubes)
|
|
|
838
|
|
|
839 60. NUTCH-581 - DistributedSearch does not update search servers added to
|
|
|
840 search-servers.txt on the fly. (Rohan Mehta via kubes)
|
|
|
841
|
|
|
842 61. NUTCH-586 - Add option to run compiled classes without job file
|
|
|
843 (enis via ab)
|
|
|
844
|
|
|
845 62. NUTCH-559 - NTLM, Basic and Digest Authentication schemes for web/proxy
|
|
|
846 server. (Susam Pal via dogacan)
|
|
|
847
|
|
|
848 63. NUTCH-534 - SegmentMerger: add -normalize option (Emmanuel Joke via ab)
|
|
|
849
|
|
|
850 64. NUTCH-528 - CrawlDbReader: add some new stats + dump into a CSV format
|
|
|
851 (Emmanuel Joke via ab)
|
|
|
852
|
|
|
853 65. NUTCH-597 - NPE in Fetcher2 (Remco Verhoef via ab)
|
|
|
854
|
|
|
855 66. NUTCH-584 - urls missing from fetchlist (Ruslan Ermilov, ab)
|
|
|
856
|
|
|
857 67. NUTCH-580 - Remove deprecated hadoop api calls (FS) (siren)
|
|
|
858
|
|
|
859 68. NUTCH-587 - Upgrade to Hadoop 0.15.3 (kubes)
|
|
|
860
|
|
|
861 69. NUTCH-604 - Upgrade to Lucene 2.3.0 (ab)
|
|
|
862
|
|
|
863 70. NUTCH-602 - Allow configurable number of handlers for search servers
|
|
|
864 (hartbecke via kubes)
|
|
|
865
|
|
|
866 71. NUTCH-607 - Update build.xml to include tika jar when building war (kubes)
|
|
|
867
|
|
|
868 72. NUTCH-608 - Upgrade nutch to use released apache-tika-0.1-incubating (mattmann)
|
|
|
869
|
|
|
870 73. NUTCH-606 - Refactoring of Generator, run all urls through checks (kubes)
|
|
|
871
|
|
|
872 74. NUTCH-605 - Change deprecated configuration methods for Hadoop (kubes)
|
|
|
873
|
|
|
874 75. NUTCH-603 - Add more default url normalizations (kubes)
|
|
|
875
|
|
|
876 76. NUTCH-611 - Upgrade Nutch to use Hadoop 0.16 (kubes)
|
|
|
877
|
|
|
878 77. NUTCH-44 - Too many search results, limits max results returned from a
|
|
|
879 single search. (Emilijan Mirceski and Susam Pal via kubes)
|
|
|
880
|
|
|
881 78. NUTCH-567 - Proper (?) handling of URIs in TagSoup. TagSoup library is
|
|
|
882 updated to 1.2 version. (dogacan)
|
|
|
883
|
|
|
884 79. NUTCH-613 - Empty summaries and cached pages (kubes via ab)
|
|
|
885
|
|
|
886 80. NUTCH-612 - URL filtering was disabled in Generator when invoked
|
|
|
887 from Crawl (Susam Pal via ab)
|
|
|
888
|
|
|
889 81. NUTCH-601 - Recrawling on existing crawl directory (Susam Pal via ab)
|
|
|
890
|
|
|
891 82. NUTCH-575 - NPE in OpenSearchServlet (John H. Lee via ab)
|
|
|
892
|
|
|
893 83. NUTCH-126 - Fetching https does not work with a proxy (Fritz Elfert via ab)
|
|
|
894
|
|
|
895 84. NUTCH-615 - Redirected URL-s fetched without setting fetchInterval.
|
|
|
896 Guard against reprUrl being null. (Emmanuel Joke, ab)
|
|
|
897
|
|
|
898 85. NUTCH-616 - Reset Fetch Retry counter when fetch is successful (Emmanuel
|
|
|
899 Joke, ab)
|
|
|
900
|
|
|
901 86. NUTCH-220 - Upgrade to PDFBox 0.7.3 (ab)
|
|
|
902
|
|
|
903 87. NUTCH-223 - Crawl.java uses Integer.MAX_VALUE (Jeff Ritchie via ab)
|
|
|
904
|
|
|
905 88. NUTCH-598 - Remove deprecated use of ToolBase. Use generics in Hadoop API.
|
|
|
906 (Emmanuel Joke, dogacan, ab)
|
|
|
907
|
|
|
908 89. NUTCH-620 - BasicURLNormalizer should collapse runs of slashes with a
|
|
|
909 single slash. (Mark DeSpain via ab)
|
|
|
910
|
|
|
911 90. NUTCH-500 - Add hadoop masters configuration file into conf folder.
|
|
|
912 (Emmanuel Joke via kubes)
|
|
|
913
|
|
|
914 91. NUTCH-596 - ParseSegments parse content even if its not
|
|
|
915 CrawlDatum.STATUS_FETCH_SUCCESS (dogacan)
|
|
|
916
|
|
|
917 92. NUTCH-618 - Tika error "Media type alias already exists" (mattmann,kubes)
|
|
|
918
|
|
|
919 93. NUTCH-634 - Upgrade Nutch to Hadoop 0.17.1 (Michael Gottesman, Lincoln
|
|
|
920 Ritter, ab)
|
|
|
921
|
|
|
922 94. NUTCH-641 - IndexSorter inorrectly copies stored fields (ab)
|
|
|
923
|
|
|
924 95. NUTCH-645 - Parse-swf unit test failing (ab)
|
|
|
925
|
|
|
926 96. NUTCH-642 - Unit tests fail when run in non-local mode (ab)
|
|
|
927
|
|
|
928 97. NUTCH-639 - Change LuceneDocumentWrapper visibility from
|
|
|
929 private to _public_ (Guillaume Smet via dogacan)
|
|
|
930
|
|
|
931 98. NUTCH-651 - Remove bin/{start|stop}-balancer.sh from svn
|
|
|
932 tracking. (dogacan)
|
|
|
933
|
|
|
934 99. NUTCH-375 - Add support for Content-Encoding: deflated
|
|
|
935 (Pascal Beis, ab)
|
|
|
936
|
|
|
937 100. NUTCH-633 - ParseSegment no longer allow reparsing.
|
|
|
938 (dogacan)
|
|
|
939
|
|
|
940 101. NUTCH-653 - Upgrade to hadoop 0.18. (dogacan)
|
|
|
941
|
|
|
942 102. NUTCH-621 - Nutch needs to declare it's crypto usage (mattmann)
|
|
|
943
|
|
|
944 103. NUTCH-654 - urlfilter-regex's main does not work.
|
|
|
945 (dogacan)
|
|
|
946
|
|
|
947 104. NUTCH-640 - confusing description "set it to Integer.MAX_VALUE".
|
|
|
948 (dogacan)
|
|
|
949
|
|
|
950 105. NUTCH-662 - Upgrade Nutch to use Lucene 2.4. (kubes)
|
|
|
951
|
|
|
952 106. NUTCH-663 - Upgrade Nutch to use Hadoop 0.19 (kubes)
|
|
|
953
|
|
|
954 107. NUTCH-647 - Resolve URLs tool (kubes)
|
|
|
955
|
|
|
956 108. NUTCH-665 - Search Load Testing Tool (kubes)
|
|
|
957
|
|
|
958 109. NUTCH-667 - Input Format for working with Content in Hadoop Streaming
|
|
|
959 (kubes)
|
|
|
960
|
|
|
961 110. NUTCH-635 - LinkAnalysis Tool for Nutch. (kubes)
|
|
|
962
|
|
|
963 111. NUTCH-646 - New Indexing Framework for Nutch. (kubes)
|
|
|
964
|
|
|
965 112. NUTCH-668 - Domain URL Filter. (kubes)
|
|
|
966
|
|
|
967 113. NUTCH-594 - Serve Nutch search results in multiple formats including
|
|
|
968 XML and JSON. (kubes)
|
|
|
969
|
|
|
970 114. NUTCH-442 - Integrate Solr/Nutch. (dogacan, original version by siren)
|
|
|
971
|
|
|
972 115. NUTCH-652 - AdaptiveFetchSchedule#setFetchSchedule doesn't calculate
|
|
|
973 fetch interval correctly. (dogacan)
|
|
|
974
|
|
|
975 116. NUTCH-627 - Minimize host address lookup (Otis Gospodnetic)
|
|
|
976
|
|
|
977 117. NUTCH-678 - Hadoop 0.19 requires an update of jets3t.
|
|
|
978 (julien nioche via dogacan)
|
|
|
979
|
|
|
980 118. NUTCH-681 - parse-mp3 compilation problem.
|
|
|
981 (Wildan Maulana via dogacan)
|
|
|
982
|
|
|
983 119. NUTCH-676 - MapWritable is written inefficiently and confusingly.
|
|
|
984 (dogacan)
|
|
|
985
|
|
|
986 120. NUTCH-579 - Feed plugin only indexes one post per feed due to identical
|
|
|
987 digest. (dogacan)
|
|
|
988
|
|
|
989 121. NUTCH-571 - parse-mp3 plugin doesn't always index album of mp3.
|
|
|
990 (Joseph Chen, dogacan)
|
|
|
991
|
|
|
992 122. NUTCH-682 - SOLR indexer does not set boost on the document.
|
|
|
993 (julien nioche via dogacan)
|
|
|
994
|
|
|
995 123. NUTCH-279 - Additions to urlnormalizer-regex (Stefan Neufeind, ab)
|
|
|
996
|
|
|
997 124. NUTCH-671 - JSP errors in Nutch searcher webapp (Edwin Chu via ab)
|
|
|
998
|
|
|
999 125. NUTCH-643 - ClassCastException in PDF parser (Guillaume Smet, ab)
|
|
|
1000
|
|
|
1001 126. NUTCH-636 - Httpclient plugin https doesn't work on IBM JRE
|
|
|
1002 (Curtis d'Entremont, ab)
|
|
|
1003
|
|
|
1004 127. NUTCH-683 - NUTCH-676 broke CrawlDbMerger. (dogacan)
|
|
|
1005
|
|
|
1006 128. NUTCH-631 - MoreIndexingFilter fails with NoSuchElementException
|
|
|
1007 (Stefan Will, siren)
|
|
|
1008
|
|
|
1009 129. NUTCH-691 - Update jakarta poi jars to the most relevant version
|
|
|
1010 (Dmitry Lihachev via siren)
|
|
|
1011
|
|
|
1012 130. NUTCH-563 - Include custom fields in BasicQueryFilter
|
|
|
1013 (Julien Nioche via siren)
|
|
|
1014
|
|
|
1015 131. NUTCH-695 - Incorrect mime type detection by MoreIndexingFilter plugin
|
|
|
1016 (Dmitry Lihachev via siren)
|
|
|
1017
|
|
|
1018 132. NUTCH-694 - Distributed Search Server fails (siren)
|
|
|
1019
|
|
|
1020 133. NUTCH-626 - Fetcher2 breaks out the domain with db.ignore.external.links
|
|
|
1021 set at cross domain redirects (Remco Verhoef, dogacan via siren)
|
|
|
1022
|
|
|
1023 134. NUTCH-247 - Robot parser to restrict (kubes, siren)
|
|
|
1024
|
|
|
1025 135. NUTCH-698 - CrawlDb is corrupted after a few crawl cycles (dogacan
|
|
|
1026 via siren)
|
|
|
1027
|
|
|
1028 136. NUTCH-699 - Add an "official" solr schema for solr integration (dogacan,
|
|
|
1029 Dmitry Lihachev via siren)
|
|
|
1030
|
|
|
1031 137. NUTCH-703 - Upgrade to Hadoop 0.19.1 (ab)
|
|
|
1032
|
|
|
1033 138. NUTCH-419 - Unavailable robots.txt kills fetch (Carsten Lehmann,
|
|
|
1034 Doug Cook via ab)
|
|
|
1035
|
|
|
1036 139. NUTCH-700 - Neko1.9.11 goes into a loop (Julien Nioche, siren)
|
|
|
1037
|
|
|
1038 140. NUTCH-669 - Consolidate code for Fetcher and Fetcher2 (siren)
|
|
|
1039
|
|
|
1040 141. NUTCH-711 - Indexer failing after upgrade to Hadoop 0.19.1 (ab)
|
|
|
1041
|
|
|
1042 142. NUTCH-684 - Dedup support for Solr. (dogacan)
|
|
|
1043
|
|
|
1044 143. NUTCH-715 - Subcollection plugin doesn't work with default
|
|
|
1045 subcollections.xml file (Dmitry Lihachev via siren)
|
|
|
1046
|
|
|
1047 144. NUTCH-722 - Nutch contains JAI jars that we cannot redistribute
|
|
|
1048
|
|
|
1049 Release 0.9 - 2007-04-02
|
|
|
1050
|
|
|
1051 1. Changed log4j confiquration to log to stdout on commandline
|
|
|
1052 tools (siren)
|
|
|
1053
|
|
|
1054 2. NUTCH-344 - Fix for thread blocking issue (Greg Kim via siren)
|
|
|
1055
|
|
|
1056 3. NUTCH-260 - Update hadoop version to 0.5.0 (Renaud Richardet,
|
|
|
1057 siren)
|
|
|
1058
|
|
|
1059 4. Optionally skip pages with abnormally large values of Crawl-Delay
|
|
|
1060 (Dennis Kubes via ab)
|
|
|
1061
|
|
|
1062 5. Change readdb -stats to use CombiningCollector (ab)
|
|
|
1063
|
|
|
1064 6. NUTCH-348 - Fix Generator to select highest scoring pages (Chris
|
|
|
1065 Schneider and Stefan Groschupf via ab)
|
|
|
1066
|
|
|
1067 7. NUTCH-347 - Adjust plugin build script not to emit warnings when copying
|
|
|
1068 dependant jars (siren)
|
|
|
1069
|
|
|
1070 8. NUTCH-338 - Remove the text parser as an option for parsing PDF files
|
|
|
1071 in parse-plugins.xml (Chris A. Mattmann via siren)
|
|
|
1072
|
|
|
1073 9. NUTCH-105 - Network error during robots.txt fetch causes file to
|
|
|
1074 be ignored (Greg Kim via siren)
|
|
|
1075
|
|
|
1076 10. NUTCH-367 - DistributedSearch thown ClassCastException (siren)
|
|
|
1077
|
|
|
1078 11. NUTCH-332 - Fix the problem of doubling scores caused by links pointing
|
|
|
1079 to the current page (e.g. anchors). (Stefan Groschupf via ab)
|
|
|
1080
|
|
|
1081 12. NUTCH-365 - Flexible URL normalization (ab)
|
|
|
1082
|
|
|
1083 13. NUTCH-336 - Differentiate between newly discovered pages and newly
|
|
|
1084 injected pages (Chris Schneider via ab) NOTE: this changes the
|
|
|
1085 scoring API, filter implementations need to be updated.
|
|
|
1086
|
|
|
1087 14. NUTCH-337 - Fetcher ignores the fetcher.parse value (Stefan Groschupf
|
|
|
1088 via ab)
|
|
|
1089
|
|
|
1090 15. NUTCH-350 - Urls blocked by http.max.delays incorrectly marked as GONE
|
|
|
1091 (Stefan Groschupf via ab)
|
|
|
1092
|
|
|
1093 16. NUTCH-374 - when http.content.limit be set to -1 and
|
|
|
1094 Response.CONTENT_ENCODING is gzip or x-gzip , it can not fetch any thing
|
|
|
1095 (King Kong via pkosiorowski)
|
|
|
1096
|
|
|
1097 17. NUTCH-383 - upgrade to Hadoop 0.7.1 and Lucene 2.0.0. (ab)
|
|
|
1098
|
|
|
1099 ****************************** WARNING !!! ********************************
|
|
|
1100 * This upgrade breaks data format compatibility. A tool 'convertdb' *
|
|
|
1101 * was added to migrate existing CrawlDb-s to the new format. Segment data *
|
|
|
1102 * can be partially migrated using 'mergesegs', however segments will *
|
|
|
1103 * require re-parsing (and consequently re-indexing). *
|
|
|
1104 ****************************** WARNING !!! ********************************
|
|
|
1105
|
|
|
1106 18. NUTCH-371 - DeleteDuplicates now correctly implements both parts of
|
|
|
1107 the algorithm. (ab)
|
|
|
1108
|
|
|
1109 19. NUTCH-391 - ParseUtil logs file contents to log file when it cannot
|
|
|
1110 find parser (siren)
|
|
|
1111
|
|
|
1112 20. NUTCH-379 - ParseUtil does not pass through the content's URL to the
|
|
|
1113 ParserFactory (Chris A. Mattmann via siren)
|
|
|
1114
|
|
|
1115 21. NUTCH-361, NUTCH-136 - When jobtracker is 'local' generate only one
|
|
|
1116 partition. (ab)
|
|
|
1117
|
|
|
1118 22. NUTCH-399 - Change CommandRunner to use concurrent api from jdk (siren)
|
|
|
1119
|
|
|
1120 23. NUTCH-395 - Increase fetching speed (siren)
|
|
|
1121
|
|
|
1122 24. NUTCH-388 - nutch-default.xml has outdated example for urlfilter.order
|
|
|
1123 (reported by Jared Dunne)
|
|
|
1124
|
|
|
1125 25. NUTCH-404 - Fix LinkDB Usage - implementation mismatch (siren)
|
|
|
1126
|
|
|
1127 26. NUTCH-403 - Make URL filtering optional in Generator (siren)
|
|
|
1128
|
|
|
1129 27. NUTCH-405 - Content object is not properly initialized in map method
|
|
|
1130 of ParseSegment (siren)
|
|
|
1131
|
|
|
1132 28. NUTCH-362 - Remove parse-text from unsupported filetypes in
|
|
|
1133 parse-plugins.xml (siren)
|
|
|
1134
|
|
|
1135 29. NUTCH-305 - Update crawl and url filter lists to exclude
|
|
|
1136 jpeg|JPEG|bmp|BMP, suffix-urlfilter.txt (contributed by Stefan
|
|
|
1137 Neufeind) is also updated (siren)
|
|
|
1138
|
|
|
1139 30. NUTCH-406 - Metadata tries to write null values (mattmann)
|
|
|
1140
|
|
|
1141 31. NUTCH-415 - Generator should mark selected records in CrawlDb.
|
|
|
1142 Due to increased resource consumption this step is optional.
|
|
|
1143 Application-level locking has been added to prevent concurrent
|
|
|
1144 modification of databases. (ab)
|
|
|
1145
|
|
|
1146 32. NUTCH-416 - CrawlDatum status and CrawlDbReducer refactoring. It is
|
|
|
1147 now possible to correctly update CrawlDb from multiple segments.
|
|
|
1148 Introduce new status codes for temporary and permanent
|
|
|
1149 redirection. (ab)
|
|
|
1150
|
|
|
1151 33. NUTCH-322 - Fix Fetcher to store redirected pages and to store
|
|
|
1152 protocol-level status. This also should fix NUTCH-273. (ab)
|
|
|
1153
|
|
|
1154 34. Change default Fetcher behavior not to follow redirects immediately.
|
|
|
1155 Instead Fetcher will record redirects as new pages to be added to CrawlDb.
|
|
|
1156 This also partially addresses NUTCH-273. (ab)
|
|
|
1157
|
|
|
1158 35. Detect and report when Generator creates 0-sized segments. (ab)
|
|
|
1159
|
|
|
1160 36. Fix Injector to preserve already existing CrawlDatum if the seed list
|
|
|
1161 being injected also contains such URL. (ab)
|
|
|
1162
|
|
|
1163 37. NUTCH-425, NUTCH-426 - Fix anchors pollution. Continue after
|
|
|
1164 skipping bad URLs. (Michael Stack via ab)
|
|
|
1165
|
|
|
1166 38. NUTCH-325 - UrlFilters.java throws NPE in case urlfilter.order contains
|
|
|
1167 Filters that are not in plugin.includes (Stefan Groschupf, siren)
|
|
|
1168
|
|
|
1169 39. NUTCH-421 - Allow predeterminate running order of indexing filters
|
|
|
1170 (Alan Tanaman, siren)
|
|
|
1171
|
|
|
1172 40. When indexing pages with redirection, drop all intermediate pages and
|
|
|
1173 index only the final page. (ab)
|
|
|
1174
|
|
|
1175 41. Upgrade to Hadoop 0.10.1. (ab)
|
|
|
1176
|
|
|
1177 42. NUTCH-420 - Fix a bug in DeleteDuplicates where results depended on the
|
|
|
1178 order in which IndexDoc-s are processed. (Dogacan Guney via ab)
|
|
|
1179
|
|
|
1180 43. NUTCH-428 - NullPointerException thrown when agent name is not
|
|
|
1181 configured properly. Changed to throw RuntimeException instead.
|
|
|
1182 (siren)
|
|
|
1183
|
|
|
1184 44. NUTCH-430 - Integer overflow in HashComparator.compare (siren)
|
|
|
1185
|
|
|
1186 45. NUTCH-68 - Add a tool to generate arbitrary fetchlists. (ab)
|
|
|
1187
|
|
|
1188 46. NUTCH-433 - java.io.EOFException in newer nightlies in mergesegs
|
|
|
1189 or indexing from hadoop.io.DataOutputBuffer (siren)
|
|
|
1190
|
|
|
1191 47. NUTCH-339 - Fetcher2: a queue-based fetcher implementation. (ab)
|
|
|
1192
|
|
|
1193 48. NUTCH-390 - Javadoc warnings (mattmann)
|
|
|
1194
|
|
|
1195 49. NUTCH-449 - Make junit output format configurable. (nigel via cutting)
|
|
|
1196
|
|
|
1197 50. NUTCH-432 - Fix a bug where platform name with spaces would break the
|
|
|
1198 bin/nutch script. (Brian Whitman via ab)
|
|
|
1199
|
|
|
1200 51. Upgrade to Hadoop 0.11.2 and Lucene 2.1.0 release. (ab)
|
|
|
1201
|
|
|
1202 52. NUTCH-167 - Observation of robots "noarchive" directive. (ab)
|
|
|
1203
|
|
|
1204 53. NUTCH-384 - Protocol-file plugin does not allow the parse plugins
|
|
|
1205 framework to operate properly (Heiko Dietze via mattmann)
|
|
|
1206
|
|
|
1207 54. NUTCH-233 - Wrong regular expression hangs reduce process forever (Stefan
|
|
|
1208 Groschupf via kubes)
|
|
|
1209
|
|
|
1210 55. NUTCH-436 - Incorrect handling of relative paths when the embedded URL
|
|
|
1211 path is empty (kubes)
|
|
|
1212
|
|
|
1213 56. Upgrade to Hadoop 0.12.1 release. (ab)
|
|
|
1214
|
|
|
1215 57. NUTCH-246 - Incorrect segment size being generated due to time
|
|
|
1216 synchronization issue (Stefan Groschupf via ab)
|
|
|
1217
|
|
|
1218 58. Upgrade to Hadoop 0.12.2 release. (ab)
|
|
|
1219
|
|
|
1220 59. NUTCH-333 - SegmentMerger and SegmentReader should use NutchJob. (Michael
|
|
|
1221 Stack and Dogacan Guney via kubes)
|
|
|
1222
|
|
|
1223 Release 0.8 - 2006-07-25
|
|
|
1224
|
|
|
1225 0. Totally new architecture, based on hadoop
|
|
|
1226 [http://lucene.apache.org/hadoop] (cutting)
|
|
|
1227
|
|
|
1228 1. NUTCH-107 - Typo in plugin/urlfilter-*/plugin.xml. (Stephen Cross).
|
|
|
1229
|
|
|
1230 2. NUTCH-108 - Log hosts that exceed generate.max.per.host.
|
|
|
1231 (Rod Taylor via cutting)
|
|
|
1232
|
|
|
1233 3. NUTCH-88 - Enhance ParserFactory plugin selection policy
|
|
|
1234 (jerome)
|
|
|
1235
|
|
|
1236 4. NUTCH-124 - Protocol-httpclient does not follow redirects when
|
|
|
1237 fetching robots.txt (cutting)
|
|
|
1238
|
|
|
1239 5. NUTCH-130 - Be explicit about target JVM when building (1.4.x?)
|
|
|
1240 (stack@archive.org, cutting)
|
|
|
1241
|
|
|
1242 6. NUTCH-114 - Getting number of urls and links from crawldb
|
|
|
1243 (Stefan Groschupf via ab)
|
|
|
1244
|
|
|
1245 7. NUTCH-112 - Link in cached.jsp page to cached content is an
|
|
|
1246 absolute link (Chris A. Mattmann via jerome)
|
|
|
1247
|
|
|
1248 8. NUTCH-135 - Http header meta data are case insensitive in the
|
|
|
1249 real world (Stefan Groschupf via jerome)
|
|
|
1250
|
|
|
1251 9. NUTCH-145 - Build of war file fails on Chinese (zh) .xml files due
|
|
|
1252 to UTF-8 BOM (KuroSaka TeruHiko via siren)
|
|
|
1253
|
|
|
1254 10. NUTCH-121 - SegmentReader for mapred (Rod Taylor via ab)
|
|
|
1255
|
|
|
1256 11. Added support for OpenSearch (cutting)
|
|
|
1257
|
|
|
1258 12. NUTCH-142 - NutchConf should use the thread context classloader
|
|
|
1259 (Mike Cannon-Brookes via pkosiorowski)
|
|
|
1260
|
|
|
1261 13. NUTCH-160 - Use standard Java Regex library rather than
|
|
|
1262 org.apache.oro.text.regex (Rod Taylor via cutting)
|
|
|
1263
|
|
|
1264 14. NUTCH-151 - CommandRunner can hang after the main thread exec is
|
|
|
1265 finished and has inefficient busy loop (Paul Baclace via cutting)
|
|
|
1266
|
|
|
1267 15. NUTCH-174 - Problem encountered with ant during compilation
|
|
|
1268
|
|
|
1269 16. NUTCH-190 - ParseUtil drops reason for failed parse
|
|
|
1270 (stack@archive.org via ab)
|
|
|
1271
|
|
|
1272 17. NUTCH-169 - Remove static NutchConf (Marko Bauhardt via ab)
|
|
|
1273
|
|
|
1274 18. NUTCH-194 - Nutch-169 introduced two tiny bugs (Marko Bauhardt via ab)
|
|
|
1275
|
|
|
1276 19. NUTCH-178 - in search.jsp must be session creation "false"
|
|
|
1277 (YourSoft via siren)
|
|
|
1278
|
|
|
1279 20. NUTCH-200 - OpenSearch Servlet ist broken
|
|
|
1280 (Marko Bauhardt via siren)
|
|
|
1281
|
|
|
1282 21. NUTCH-81 - Webapp only works when deployed in root
|
|
|
1283 (AJ Banck, Michael Nebel via siren)
|
|
|
1284
|
|
|
1285 22. NUTCH-139 - Standard metadata property names in the ParseData
|
|
|
1286 metadata (Chris A. Mattmann, jerome)
|
|
|
1287
|
|
|
1288 23. NUTCH-192 - Meta data support for CrawlDatum
|
|
|
1289 (Stefan Groschupf via ab)
|
|
|
1290
|
|
|
1291 24. NUTCH-52 - Parser plugin for MS Excel files
|
|
|
1292 (Rohit Kulkarni via jerome)
|
|
|
1293
|
|
|
1294 25. NUTCH-53 - Parser plugin for Zip files
|
|
|
1295 (Rohit Kulkarni via jerome)
|
|
|
1296
|
|
|
1297 26. NUTCH-137 - footer is not displayed in search result page
|
|
|
1298 (KuroSaka TeruHiko via siren)
|
|
|
1299
|
|
|
1300 27. NUTCH-118 - FAQ link points to invalid URL
|
|
|
1301 (Steve Betts via siren)
|
|
|
1302
|
|
|
1303 28. NUTCH-184 - Serbian (sr, Cyrilic) and Serbo-Croatian (sh, Latin)
|
|
|
1304 translation (Ivan Sekulovic via siren)
|
|
|
1305
|
|
|
1306 29. NUTCH-211 - FetchedSegments leave readers open (Stefan Groschupf
|
|
|
1307 via cutting)
|
|
|
1308
|
|
|
1309 30. NUTCH-140 - Add alias capability in parse-plugins.xml file that
|
|
|
1310 allows mimeType->extensionId mapping (Chris A. Mattmann via jerome)
|
|
|
1311
|
|
|
1312 31. NUTCH-214 - Added Links to web site to search mailling list
|
|
|
1313 (Jake Vanderdray via jerome)
|
|
|
1314
|
|
|
1315 32. NUTCH-204 - Multiple field values in HitDetails
|
|
|
1316 (Stefan Groschupf via jerome)
|
|
|
1317
|
|
|
1318 33. NUTCH-219 - file.content.limit & ftp.content.limit should be changed
|
|
|
1319 to -1 to be consistent with http (jerome)
|
|
|
1320
|
|
|
1321 34. NUTCH-221 - Prepare nutch for upcoming lucene 2.0 (siren)
|
|
|
1322
|
|
|
1323 35. NUTCH-91 - Empty encoding causes exception (Michael Nebel via
|
|
|
1324 pkosiorowski)
|
|
|
1325
|
|
|
1326 36. NUTCH-228 - Clustering plugin descriptor broken (Dawid Weiss via
|
|
|
1327 jerome)
|
|
|
1328
|
|
|
1329 37. NUTCH-229 - Improved handling of plugin folder configuration
|
|
|
1330 (Stefan Groschupf via ab)
|
|
|
1331
|
|
|
1332 38. NUTCH-206 - Search server throws InstantiationException (ab)
|
|
|
1333
|
|
|
1334 39. NUTCH-203 - ParseSegment throws InstantiationException (Marko Bauhardt
|
|
|
1335 via ab)
|
|
|
1336
|
|
|
1337 40. NUTCH-3 - Multi values of header discarded (Stefan Groschupf via ab)
|
|
|
1338
|
|
|
1339 41. Update to lucene 1.9.1 (cutting)
|
|
|
1340
|
|
|
1341 42. NUTCH-235 - Duplicate Inlink values (ab)
|
|
|
1342
|
|
|
1343 43. NUTCH-234 - Clustering extension code cleanups and a real
|
|
|
1344 JUnit test case for the current implementation (Dawid Weiss via ab)
|
|
|
1345
|
|
|
1346 44. NUTCH-210 - Context.xml file for Nutch web application
|
|
|
1347 (Chris A. Mattmann via jerome)
|
|
|
1348
|
|
|
1349 45. NUTCH-231 - Invalid CSS entries (AJ Banck via jerome)
|
|
|
1350
|
|
|
1351 46. NUTCH-232 - Search.jsp has multiple search forms creating
|
|
|
1352 invalid html / incorrect focus function (jerome)
|
|
|
1353
|
|
|
1354 47. NUTCH-196 - lib-xml and lib-log4j plugins (ab, jerome)
|
|
|
1355
|
|
|
1356 48. NUTCH-244 - Inconsistent handling of property values
|
|
|
1357 boundaries / unable to set db.max.outlinks.per.page to
|
|
|
1358 infinite (jerome)
|
|
|
1359
|
|
|
1360 49. NUTCH-245 - DTD for plugin.xml configuration files
|
|
|
1361 (Chris A. Mattmann via jerome)
|
|
|
1362
|
|
|
1363 50. NUTCH-250 - Generate to log truncation caused by
|
|
|
1364 generate.max.per.host (Rod Taylor via cutting)
|
|
|
1365
|
|
|
1366 51. NUTCH-125 - OpenOffice Parser plugin (ab)
|
|
|
1367
|
|
|
1368 52. Switch from using java.io.File to org.apache.hadoop.fs.Path.
|
|
|
1369 (cutting)
|
|
|
1370
|
|
|
1371 53. NUTCH-240 - Scoring API: extension point, scoring filters and
|
|
|
1372 an OPIC plugin (ab)
|
|
|
1373
|
|
|
1374 54. NUTCH-134 - Summarizer doesn't select the best snippets (jerome)
|
|
|
1375
|
|
|
1376 55. NUTCH-268 - Generator and lib-http use different definitions of
|
|
|
1377 "unique host" (ab)
|
|
|
1378
|
|
|
1379 56. NUTCH-280 - Url query causes NullPointerException (Grant Glouser
|
|
|
1380 via siren)
|
|
|
1381
|
|
|
1382 57. NUTCH-285 - LinkDb Fails rename doesn't create parent directories
|
|
|
1383 (Dennis Kubes via ab)
|
|
|
1384
|
|
|
1385 58. NUTCH-201 - Add support for subcollections
|
|
|
1386 (siren)
|
|
|
1387
|
|
|
1388 59. NUTCH-298 - If a 404 for a robots.txt is returned a NPE is thrown
|
|
|
1389 (Stefan Groschupf via jerome)
|
|
|
1390
|
|
|
1391 60. NUTCH-275 - Fetcher not parsing XHTML-pages at all (jerome)
|
|
|
1392
|
|
|
1393 61. NUTCH-301 - CommonGrams loads analysis.common.terms.file for each query
|
|
|
1394 (Stefan Groschupf via jerome)
|
|
|
1395
|
|
|
1396 62. NUTCH-110 - OpenSearchServlet outputs illegal xml characters
|
|
|
1397 (stack@archive.org via siren)
|
|
|
1398
|
|
|
1399 63. NUTCH-292 - OpenSearchServlet: OutOfMemoryError: Java heap space
|
|
|
1400 (Stefan Neufeind via siren)
|
|
|
1401
|
|
|
1402 64. NUTCH-307 - Wrong configured log4j.properties (jerome)
|
|
|
1403
|
|
|
1404 65. NUTCH-303 - Logging improvements (jerome)
|
|
|
1405
|
|
|
1406 66. NUTCH-308 - Maximum search time limit (ab)
|
|
|
1407
|
|
|
1408 67. NUTCH-306 - DistributedSearch.Client liveAddresses concurrency
|
|
|
1409 problem (Grant Glouser via siren)
|
|
|
1410
|
|
|
1411 68. Update to hadoop-0.4 (Milind Bhandarkar, cutting)
|
|
|
1412
|
|
|
1413 69. NUTCH-317 - Clarify what the queryLanguage argument of
|
|
|
1414 Query.parse(...) means (jerome)
|
|
|
1415
|
|
|
1416 70. Added alternative experimental web gui in contrib containing
|
|
|
1417 extensions like subcollection, keymatch, user preferences,
|
|
|
1418 caching, implemented mainly using tiles and jstl (siren)
|
|
|
1419
|
|
|
1420 71. NUTCH-320 DmozParser does not output list of urls to stdout
|
|
|
1421 but to a log file instead. Original functionality restored.
|
|
|
1422
|
|
|
1423 72. NUTCH-271 - Add ability to limit crawling to the set of initially
|
|
|
1424 injected hosts (db.ignore.external.links) (Philippe Eugene,
|
|
|
1425 Stefan Neufeind via ab)
|
|
|
1426
|
|
|
1427 73. NUTCH-293 - Support for Crawl-Delay (Stefan Groschupf via ab)
|
|
|
1428
|
|
|
1429 74. NUTCH-327 - Fixed logging directory on cygwin (siren)
|
|
|
1430
|
|
|
1431 Release 0.7 - 2005-08-17
|
|
|
1432
|
|
|
1433 1. Added support for "type:" in queries. Search results are limited/qualified
|
|
|
1434 by mimetype or its primary type or sub type. For example,
|
|
|
1435 (1) searching with "type:application/pdf" restricts results
|
|
|
1436 to pages which were identified to be of mimetype "application/pdf".
|
|
|
1437 (2) with "type:application", nutch will return pages of
|
|
|
1438 primary type "application".
|
|
|
1439 (3) with "type:pdf", only pages of sub type "pdf" will be listed.
|
|
|
1440 (John Xing, 20050120)
|
|
|
1441
|
|
|
1442 2. Added support for "date:" in queries. Last-Modified is indexed.
|
|
|
1443 Search results are restricted by lower and upper date (inclusive)
|
|
|
1444 as date:yyyymmdd-yyyymmdd. For example, date:20040101-20041231
|
|
|
1445 only returns pages with Last-Modified in year 2004.
|
|
|
1446 (John Xing, 20050122)
|
|
|
1447
|
|
|
1448 3. Add URLFilter plugin interface and convert existing url filters into
|
|
|
1449 plugins. (John Xing, 20050206)
|
|
|
1450
|
|
|
1451 4. Add UpdateSegmentsFromDb tool, which updates the scores and
|
|
|
1452 anchors of existing segments with the current values in the web
|
|
|
1453 db. This is used by CrawlTool, so that pages are now only fetched
|
|
|
1454 once per crawl. (Doug Cutting, 20050221)
|
|
|
1455
|
|
|
1456 5. Moved code into org.apache.nutch sub-packages. Changed license to
|
|
|
1457 Apache 2.0. Removed jar files whose licenses do not permit
|
|
|
1458 redistribution by Apache. Disabled compilation of plugins which
|
|
|
1459 require these libraries. (Doug Cutting 20050301)
|
|
|
1460
|
|
|
1461 6. Index host and title in separate fields. Host was indexed
|
|
|
1462 previously only as a part of the URL. Title was indexed as an
|
|
|
1463 anchor. Now boosts for matching these fields may be adjusted
|
|
|
1464 separately from boosts for matching anchors and url. Also: move
|
|
|
1465 site indexing to index-basic plugin to minimize the number of
|
|
|
1466 times the URL needs to be parsed; and, stop using anchor analyzer
|
|
|
1467 for anything but anchors. (Piotr Kosiorowski via Doug Cutting
|
|
|
1468 20050323)
|
|
|
1469
|
|
|
1470 7. Add servlet Cached.java that serves cached Content of any mime type.
|
|
|
1471 Slightly modified are web.xml and cached.jsp.
|
|
|
1472 (John Xing, 20050401)
|
|
|
1473
|
|
|
1474 8. Add skipCompressedByteArray() to WritableUtils.java.
|
|
|
1475 (John Xing, 20050402)
|
|
|
1476
|
|
|
1477 9. Fixes to jsp and static web pages. These now use relative links,
|
|
|
1478 so that the Nutch webapp file can be used in places other than at
|
|
|
1479 the root. Also fixed links to the about and help pages. Bug #32.
|
|
|
1480 (Jerome Charron via cutting, 20050404)
|
|
|
1481
|
|
|
1482 10. Added some features to DistributedSearch: new segments can be added
|
|
|
1483 to searchservers without restarting the frontend, defective search
|
|
|
1484 servers are not queried until tey come back online, watchdog keeps
|
|
|
1485 an eye for your searchservers and writes simple statistics.
|
|
|
1486 (Sami Siren, 20050407)
|
|
|
1487
|
|
|
1488 11. Fix for bug #4 - Unbalanced quote in query eats all resources.
|
|
|
1489 (Piotr Kosiorowski, Sami Siren, 20050407)
|
|
|
1490
|
|
|
1491 12. Close Issue #33 - MIME content type detector (using magic char sequences).
|
|
|
1492 (Jerome Charron and Hari Kodungallur via John Xing, 20050416)
|
|
|
1493
|
|
|
1494 13. Add a servlet that implements A9's OpenSearch RSS web service.
|
|
|
1495 (cutting, 20050418)
|
|
|
1496
|
|
|
1497 14. Remove references to link analysis from tutorial, and enable
|
|
|
1498 scoring by link count when generating fetchlists and searching.
|
|
|
1499 (cutting, 20040419)
|
|
|
1500
|
|
|
1501 15. Make query boosts for host, title, anchor and phrase matches
|
|
|
1502 configurable. (Piotr Kosiorowski via cutting, 20050419)
|
|
|
1503
|
|
|
1504 16. Add support for sorting search results and search-time deduping by
|
|
|
1505 fields other than site.
|
|
|
1506
|
|
|
1507 17. Automatically convert range queries into cached range filters.
|
|
|
1508 This improves the performance and scalability of, e.g., date range
|
|
|
1509 searching.
|
|
|
1510
|
|
|
1511 18. Several methods have been renamed due to misspellings. The old
|
|
|
1512 methods have been deprecated and will be removed before the 1.0
|
|
|
1513 release.
|
|
|
1514
|
|
|
1515
|
|
|
1516 Release 0.6
|
|
|
1517
|
|
|
1518 1. Added clustering-carrot2 plugin, together with introduction of clustering
|
|
|
1519 api and modification to search jsp. (Dawid Weiss via John Xing, 20040809)
|
|
|
1520
|
|
|
1521 2. Make a number of changes to NDFS (Nutch Distributed File System)
|
|
|
1522 to fix bugs, add admin tools, etc.
|
|
|
1523
|
|
|
1524 Also, modify all command line tools so you can indicate whether to
|
|
|
1525 use NDFS or the local filesystem. If you indicate nothing, then
|
|
|
1526 it defaults to the local fs.
|
|
|
1527
|
|
|
1528 I've used this to do a 35m page crawl via NDFS, distributed over a
|
|
|
1529 dozen machines. (Mike Cafarella)
|
|
|
1530
|
|
|
1531 3. Add support for BASE tags in HTML. Outlinks are now correctly
|
|
|
1532 extracted when a BASE tag is present. (cutting)
|
|
|
1533
|
|
|
1534 4. Fix two bugs in result pagination. When the last hit on a page
|
|
|
1535 was the last hit overall, the "next" button was sometimes shown
|
|
|
1536 when the "show all" button should be shown instead. Also, in
|
|
|
1537 certain cases, the "show all" button would be shown when the
|
|
|
1538 "next" button should have been shown. (cutting)
|
|
|
1539
|
|
|
1540 5. Add config parameter "indexer.max.tokens" that determines the
|
|
|
1541 maximum number of tokens indexed per field. (Andy Hedges via cutting)
|
|
|
1542
|
|
|
1543 6. Add parser for mp3 files. (Andy Hedges via cutting)
|
|
|
1544
|
|
|
1545 7. Add RegexUrlNormalizer. This is useful for things like stripping
|
|
|
1546 out session IDs from URLs. To use it, add values for
|
|
|
1547 urlnormalizer.class and urlnormalizer.regex.file to your
|
|
|
1548 nutch-site.xml. The RegexUrlNormalizer class extends the
|
|
|
1549 BasicUrlNormalizer, and does basic normalization as well.
|
|
|
1550 (Luke Baker via cutting)
|
|
|
1551
|
|
|
1552 8. Added Swedish translation (Stefan Verzel via Sami Siren, 20040910)
|
|
|
1553
|
|
|
1554 9. Added Polish translation (Andrzej Bialecki, 20040911)
|
|
|
1555
|
|
|
1556 10. Added 3 more language profiles to language identifier (ru,hu,pl).
|
|
|
1557 Other changes to language identifier: Porfiles converted to utf8,
|
|
|
1558 added some test cases, changed the similarity calculation.
|
|
|
1559 (Sami Siren, 20040925)
|
|
|
1560
|
|
|
1561 11. Added plugin parse-rtf (Andy Hedges via John Xing, 20040929)
|
|
|
1562
|
|
|
1563 12. Added plugin index-more and more.jsp (John Xing, 20041003)
|
|
|
1564
|
|
|
1565 13. Added "View as Plain Text" feature. A new op OP_PARSETEXT is introduced
|
|
|
1566 in DistributedSearch.java. text.jsp is added. (John Xing, 20041006)
|
|
|
1567
|
|
|
1568 14. Fixed a bug that fails cached.jsp, explain.jsp, anchors.jsp and text.jsp
|
|
|
1569 (but not search.jsp) with NullPointerException in distributed search.
|
|
|
1570 It seems that this bug appears after "hits per site" stuff is added.
|
|
|
1571 The fix is done in Hit.java, making sure String site is never null.
|
|
|
1572 Hope this fix not have bad effetct on "hits per site" code.
|
|
|
1573 (John Xing, 20041006)
|
|
|
1574
|
|
|
1575 15. Fixed a bug that fails fullyDelete() in FileUtil.java for
|
|
|
1576 LocalFileSystem.java. This bug also exposes possible incompleteness
|
|
|
1577 of NDFSFile.java, where a few methods are not supported, including
|
|
|
1578 delete(). Nothing changed in NDFSFile.java though. Leave it for future
|
|
|
1579 improvement (John Xing, 20041022).
|
|
|
1580
|
|
|
1581 16. Introduced option -noParsing to Fetcher.java and added ParseSegment.java.
|
|
|
1582 A new status code CANT_PARSE is added to FetcherOutput.java.
|
|
|
1583 Without option -noParsing , no change in fetcher behavior. With
|
|
|
1584 option -noParsing, fetcher does crawls only, no parsing is carried out.
|
|
|
1585 Then, ParseSegment.java should be used to parse in separate pass.
|
|
|
1586 (John Xing, 20041025)
|
|
|
1587
|
|
|
1588 17. Added ontology plugin. Currently it is used for query refinement, as
|
|
|
1589 examplified in refine-query-init.jsp and refine-query.jsp. By default,
|
|
|
1590 query refinement is disabled in search.jsp. Please check
|
|
|
1591 ./src/plugin/ontology/README.txt for further description.
|
|
|
1592 Ontology plugin certainly can be used for many other things.
|
|
|
1593 (Michael J. Pan via John Xing, 20041129)
|
|
|
1594
|
|
|
1595 18. Changed fetcher.server.delay to be a float, so that sub-second
|
|
|
1596 delays can be specified. (cutting)
|
|
|
1597
|
|
|
1598 19. Added plugin.includes config parameter that determines which
|
|
|
1599 plugins are included. By default now only http, html and basic
|
|
|
1600 indexing and search plugins are enabled, rather than all plugins.
|
|
|
1601 This should make default performance more predictable and reliable
|
|
|
1602 going forward. (cutting)
|
|
|
1603
|
|
|
1604 20. Cleaned up some filesystem code, including:
|
|
|
1605
|
|
|
1606 - Replaced BufferedRandomAccessFile with two simpler utilties,
|
|
|
1607 NFSDataInputStream and NFSDataOutputStream.
|
|
|
1608
|
|
|
1609 - Fixed the bug where SequenceFiles were no longer flushed when
|
|
|
1610 created, so that, when fetches crashed, segments were
|
|
|
1611 unreadable. Now segments are always readable after crashes.
|
|
|
1612 Only the contents of the last buffer is lost.
|
|
|
1613
|
|
|
1614 - Simplified the FSOutputStream API to not include seek(). We
|
|
|
1615 should never need that functionality.
|
|
|
1616
|
|
|
1617 - Simplified LocalFileSystem's implementations of FSInputStream
|
|
|
1618 and FSOutputStream and optimized FSInputStream.seek().
|
|
|
1619
|
|
|
1620 (cutting)
|
|
|
1621
|
|
|
1622 21. Fixed BasicUrlNormalizer to better handle relative urls. The file
|
|
|
1623 part of a URL is normalized in the following manner:
|
|
|
1624
|
|
|
1625 1. "/aa/../" will be replaced by "/" This is done step by step until
|
|
|
1626 the url doesn´t change anymore. So we ensure, that
|
|
|
1627 "/aa/bb/../../" will be replaced by "/", too
|
|
|
1628
|
|
|
1629 2. leading "/../" will be replaced by "/"
|
|
|
1630
|
|
|
1631 (Sven Wende via cutting)
|
|
|
1632
|
|
|
1633 22. Fix Page constructors so that next fetch date is less likely to be
|
|
|
1634 misconstrued as a float. This patches a problem in WebDBInjector,
|
|
|
1635 where new pages were added to the db with nextScore set to the
|
|
|
1636 intended nextFetch date. This, in turn, confused link analysis.
|
|
|
1637
|
|
|
1638 23. In ndfs code, replace addLocalFile(), putToLocalFile() with
|
|
|
1639 copyFromLocalFile(), moveFromLocalFile(), copyToLocalFile() and
|
|
|
1640 moveToLocalFile(). (John Xing, 20041217)
|
|
|
1641
|
|
|
1642 24. Added new config parameter fetcher.threads.per.host. This is used
|
|
|
1643 by the Http protocol. When this is one behavior is as before.
|
|
|
1644 When this is greater than one then multiple threads are permitted
|
|
|
1645 to access a host at once. Note that fetcher.server.delay is no
|
|
|
1646 longer consistently observed when this is greater than one.
|
|
|
1647 (Luke Baker via Doug Cutting)
|
|
|
1648
|
|
|
1649 Release 0.5
|
|
|
1650
|
|
|
1651 1. Changed plugin directory to be a list of directories.
|
|
|
1652
|
|
|
1653 2. Permit Plugin to be the default plugin implementation.
|
|
|
1654
|
|
|
1655 3. Added pluggable interface for network protocols in new package
|
|
|
1656 net.nutch.protocol. Moved http code from core into a plugin.
|
|
|
1657
|
|
|
1658 4. Added pluggable interface for content parsing in new package
|
|
|
1659 net.nutch.parse. Moved html parsing code from core into a
|
|
|
1660 plugin.
|
|
|
1661
|
|
|
1662 5. Fixed a bug in NutchAnalysis where 16-bit characters were not
|
|
|
1663 processed correctly.
|
|
|
1664
|
|
|
1665 6. Fixed bug #971731: random summaries on result page.
|
|
|
1666 (Daniel Naber via cutting)
|
|
|
1667
|
|
|
1668 7. Made Nutch logo transparent. (Daniel Naber via cutting)
|
|
|
1669
|
|
|
1670 8. Added file protocol plugin. (John Xing via cutting)
|
|
|
1671
|
|
|
1672 9. Added ftp protocol plugin. (John Xing via cutting)
|
|
|
1673
|
|
|
1674 10. Added pdf and msword parser plugins. (John Xing via cutting)
|
|
|
1675
|
|
|
1676 11. Added pluggable indexing interface. By default, url, content,
|
|
|
1677 anchors and title are indexed, as before, but now one can easily
|
|
|
1678 alter this to, e.g., index metadata. A demonstration is provided
|
|
|
1679 which extracts and indexes Creative Commons license urls. (cutting)
|
|
|
1680
|
|
|
1681 12. Add language identification plugin.
|
|
|
1682
|
|
|
1683 The process of identification is as follows:
|
|
|
1684
|
|
|
1685 1. html (html only, HTML 4.0 "lang" attribute)
|
|
|
1686 2. meta tags (html only, http-equiv, dc.language)
|
|
|
1687 3. http header (Content-Language)
|
|
|
1688 4. if all above fail "statistical analysis"
|
|
|
1689
|
|
|
1690 1 & 2 are run during the fetching phase and 3 & 4 are run on
|
|
|
1691 indexing phase.
|
|
|
1692
|
|
|
1693 Currently supported languages (in "statistical analysis") are
|
|
|
1694 da,de,el,en,es,fi,fr,it,nl,sv and pt. The corpus used was grabbed
|
|
|
1695 from http://www.isi.edu/~koehn/europarl/ and the profiles were
|
|
|
1696 build with tool supplied in patch.
|
|
|
1697
|
|
|
1698 After indexing the language can be found from field named "lang"
|
|
|
1699
|
|
|
1700 It's not 100% accurate but it's a start.
|
|
|
1701 (Sami Siren)
|
|
|
1702
|
|
|
1703 13. Added SegmentMergeTool and "mergesegs" command, to remove
|
|
|
1704 duplicated or otherwise not used content from several segments and
|
|
|
1705 joining them together into a single new segment. The tool also
|
|
|
1706 optionally performs several other steps required for proper
|
|
|
1707 operation of Nutch - such as indexing segments, deleting
|
|
|
1708 duplicates, merging indices, and indexing the new single segment.
|
|
|
1709 (Andrzej Bialecki)
|
|
|
1710
|
|
|
1711 14. Add the ability to retrieve ParseData of a search hit. ParseData
|
|
|
1712 contains many valuable properties of a search hit.
|
|
|
1713
|
|
|
1714 This is required (among others) to properly display the cached
|
|
|
1715 content because it's not possible to determine the character
|
|
|
1716 encoding from the output of the getContent() method (which returns
|
|
|
1717 byte[]). The symptoms are that for HTML pages using non-latin1 or
|
|
|
1718 non-UTF8 encodings the cached preview will almost certainly look
|
|
|
1719 broken. Using the attached patch it is possible to determine the
|
|
|
1720 character encoding from the ParseData (for HTTP: Content-Type
|
|
|
1721 metadata), and encode the content accordingly. (Andrzej Bialecki)
|
|
|
1722
|
|
|
1723 15. Add a pluggable query interface. By default, the content, anchor
|
|
|
1724 and url fields are searched as before. A sample plugin indexes
|
|
|
1725 the host name and adds a "site:" keyword to query parsing.
|
|
|
1726
|
|
|
1727 16. Added support for "lang:" in queries. For example, searching with
|
|
|
1728 "lang:en" restricts results to pages which were identified to
|
|
|
1729 be in English.
|
|
|
1730
|
|
|
1731 17. Automatically optimize field queries to use cached Lucene filters.
|
|
|
1732 This makes, for example, searches restricted by languages or sites
|
|
|
1733 that are very common much faster.
|
|
|
1734
|
|
|
1735 18. Improved charset handling in jsp pages. (jshin by cutting)
|
|
|
1736
|
|
|
1737 19. Permit topic filtering when injecting DMOZ pages. (jshin by cutting)
|
|
|
1738
|
|
|
1739 20. When parsing crawled pages, interpret charset specifications in
|
|
|
1740 html meta tags. (jshin by cutting)
|
|
|
1741
|
|
|
1742 21. Added support for "cc:licensed" in queries, which searches for documents
|
|
|
1743 released under Creative Commons licenses. Attributes of the
|
|
|
1744 license may also be queried, with, e.g., "cc:by" for
|
|
|
1745 attribution-required licenses, "cc:nc" for non-commercial
|
|
|
1746 licenses, etc.
|
|
|
1747
|
|
|
1748 22. Relative paths named in plugin.folders are now searched for on the
|
|
|
1749 classpath. This makes, e.g., deployment in a war file much simpler.
|
|
|
1750
|
|
|
1751 23. Modifications to Fetcher.java.
|
|
|
1752
|
|
|
1753 1. Make sure it works properly with regard to creation and initialization
|
|
|
1754 of plugin instances. The problem was that multiple threads race to
|
|
|
1755 startUp() or shutDown() plugin instances. It was solved by synchronizing
|
|
|
1756 certain codes in PluginRepository.java and Extension.java.
|
|
|
1757 (Stefan Groschupf via John Xing)
|
|
|
1758
|
|
|
1759 2. Added code to explictly shutDown() plugins. Otherwise FetcherThreads
|
|
|
1760 may never return (quit) if there are still data or other structures
|
|
|
1761 (e.g., persistent socket connections) associated with plugins. (John Xing)
|
|
|
1762
|
|
|
1763 3. Fixed one type of Fetcher "hang" problems by monitoring named
|
|
|
1764 FetcherThreads. If all FetcherThreads are gone (finished),
|
|
|
1765 Fetcher.java is considered done. The problem was: there could be
|
|
|
1766 runaway threads started by external libs via FetcherThreads.
|
|
|
1767 Those threads never return, thus keep Fetcher from exiting normally.
|
|
|
1768 (John Xing)
|
|
|
1769
|
|
|
1770 24. Eliminate excessive hits from sites. This is done efficiently by
|
|
|
1771 adding the site name to Hit instances, and, when needed,
|
|
|
1772 re-querying with too-frequent sites prohibited in the query.
|
|
|
1773
|
|
|
1774
|
|
|
1775 Release 0.4
|
|
|
1776
|
|
|
1777 1. Http class refactored. (Kevin Smith via Tom Pierce)
|
|
|
1778
|
|
|
1779 2. Add Finnish translation. (Sampo Syreeni via Doug Cutting)
|
|
|
1780
|
|
|
1781 3. Added Japanese translation. (Yukio Andoh via Doug Cutting)
|
|
|
1782
|
|
|
1783 4. Updated Dutch translation. (Ype Kingma via Doug Cutting)
|
|
|
1784
|
|
|
1785 5. Initial version of Distributed DB code. (Mike Cafarella)
|
|
|
1786
|
|
|
1787 6. Make things more tolerant of crashed fetcher output files.
|
|
|
1788 (Doug Cutting)
|
|
|
1789
|
|
|
1790 7. New skin for website. (Frank Henze via Doug Cutting)
|
|
|
1791
|
|
|
1792 8. Added Spanish translation. (Diego Basch via Doug Cutting)
|
|
|
1793
|
|
|
1794 9. Add FTP support to fetcher. (John Xing via Doug Cutting)
|
|
|
1795
|
|
|
1796 10. Added Thai translation. (Pichai Ongvasith via Doug Cutting)
|
|
|
1797
|
|
|
1798 11. Added Robots.txt & throttling support to Fetcher.java. (Mike
|
|
|
1799 Cafarella)
|
|
|
1800
|
|
|
1801 12. Added nightly build. (Doug Cutting)
|
|
|
1802
|
|
|
1803 13. Default all link scores to 1.0. (Doug Cutting)
|
|
|
1804
|
|
|
1805 14. Permit one to keep internal links. (Doug Cutting)
|
|
|
1806
|
|
|
1807 15. Fixed dedup to select shortest URL. (Doug Cutting)
|
|
|
1808
|
|
|
1809 16. Changed index merger so that merged index is written to named
|
|
|
1810 directory, rather than to a generated name in that directory.
|
|
|
1811 (Doug Cutting)
|
|
|
1812
|
|
|
1813 17. Disable coordination weighting of query clauses and other minor
|
|
|
1814 scoring improvements. (Doug Cutting)
|
|
|
1815
|
|
|
1816 18. Added a new command, crawl, that constructs a database, injects a
|
|
|
1817 url file and performs a few rounds of generate/fetch/updatedb.
|
|
|
1818 This simplifies use for intranet sites. Changed some defaults to
|
|
|
1819 be more intranet friendly. (Doug Cutting)
|
|
|
1820
|
|
|
1821 19. Fixed a bug where Fetcher.java didn't construct correct relative
|
|
|
1822 links when a page was redirected. (Doug Cutting)
|
|
|
1823
|
|
|
1824 20. Fixed a query parser problem with lookahead over plusses and minuses.
|
|
|
1825 (Doug Cutting)
|
|
|
1826
|
|
|
1827 21. Add support for HTTP proxy servers. (Sami Siren via Doug Cutting)
|
|
|
1828
|
|
|
1829 22. Permit searching while fetching and/or indexing.
|
|
|
1830 (Sami Siren via Doug Cutting)
|
|
|
1831
|
|
|
1832 23. Fix a bug when throttling is disabled. (Sami Siren via Doug Cutting)
|
|
|
1833
|
|
|
1834 24. Updated Bahasa Malaysia translation. (Michael Lim via Doug Cutting)
|
|
|
1835
|
|
|
1836 25. Added Catalan translation. (Xavier Guardiola via Doug Cutting)
|
|
|
1837
|
|
|
1838 26. Added brazilian portuguese translation.
|
|
|
1839 (A. Moreir via Doug Cutting)
|
|
|
1840
|
|
|
1841 27. Added a french translation. (Julien Nioche via Doug Cutting)
|
|
|
1842
|
|
|
1843 28. Updated to Lucene 1.4RC3. (Doug Cutting)
|
|
|
1844
|
|
|
1845 29. Add capability to boost by link count & use it in crawl tool.
|
|
|
1846 (Doug Cutting)
|
|
|
1847
|
|
|
1848 30. Added plugin system. (Stefan Groschupf via Doug Cutting)
|
|
|
1849
|
|
|
1850 31. Add this change log file, for recording significant changes to
|
|
|
1851 Nutch. Populate it with changes from the last few months.
|
