binary: rejigger ripgrep's handling of binary files

This commit attempts to surface binary filtering in a slightly more
user friendly way. Namely, before, ripgrep would silently stop
searching a file if it detected a NUL byte, even if it had previously
printed a match. This can lead to the user quite reasonably assuming
that there are no more matches, since a partial search is fairly
unintuitive. (ripgrep has this behavior by default because it really
wants to NOT search binary files at all, just like it doesn't search
gitignored or hidden files.)

With this commit, if a match has already been printed and ripgrep detects
a NUL byte, then it will print a warning message indicating that the search
stopped prematurely.

Moreover, this commit adds a new flag, --binary, which causes ripgrep to
stop filtering binary files, but in a way that still avoids dumping
binary data into terminals. That is, the --binary flag makes ripgrep
behave more like grep's default behavior.

For files explicitly specified in a search, e.g., `rg foo some-file`,
then no binary filtering is applied (just like no gitignore and no
hidden file filtering is applied). Instead, ripgrep behaves as if you
gave the --binary flag for all explicitly given files.

This was a fairly invasive change, and potentially increases the UX
complexity of ripgrep around binary files. (Before, there were two
binary modes, where as now there are three.) However, ripgrep is now a
bit louder with warning messages when binary file detection might
otherwise be hiding potential matches, so hopefully this is a net
improvement.

Finally, the `-uuu` convenience now maps to `--no-ignore --hidden
--binary`, since this is closer to the actualy intent of the
`--unrestricted` flag, i.e., to reduce ripgrep's smart filtering. As a
consequence, `rg -uuu foo` should now search roughly the same number of
bytes as `grep -r foo`, and `rg -uuua foo` should search roughly the
same number of bytes as `grep -ra foo`. (The "roughly" weasel word is
used because grep's and ripgrep's binary file detection might differ
somewhat---perhaps based on buffer sizes---which can impact exactly what
is and isn't searched.)

See the numerous tests in tests/binary.rs for intended behavior.

Fixes #306, Fixes #855

7 files changed, 850 insertions, 9 deletions

diff --git a/tests/binary.rs b/tests/binary.rs
new file mode 100644
index 00000000..d2640988
--- /dev/null
+++ b/tests/binary.rs
@@ -0,0 +1,315 @@
+use crate::util::{Dir, TestCommand};
+
+// This file contains a smattering of tests specifically for checking ripgrep's
+// handling of binary files. There's quite a bit of discussion on this in this
+// bug report: https://github.com/BurntSushi/ripgrep/issues/306
+
+// Our haystack is the first 500 lines of Gutenberg's copy of "A Study in
+// Scarlet," with a NUL byte at line 237: `abcdef\x00`.
+//
+// The position and size of the haystack is, unfortunately, significant. In
+// particular, the NUL byte is specifically inserted at some point *after* the
+// first 8192 bytes, which corresponds to the initial capacity of the buffer
+// that ripgrep uses to read files. (grep for DEFAULT_BUFFER_CAPACITY.) The
+// position of the NUL byte ensures that we can execute some search on the
+// initial buffer contents without ever detecting any binary data. Moreover,
+// when using a memory map for searching, only the first 8192 bytes are
+// scanned for a NUL byte, so no binary bytes are detected at all when using
+// a memory map (unless our query matches line 237).
+//
+// One last note: in the tests below, we use --no-mmap heavily because binary
+// detection with memory maps is a bit different. Namely, NUL bytes are only
+// searched for in the first few KB of the file and in a match. Normally, NUL
+// bytes are searched for everywhere.
+//
+// TODO: Add tests for binary file detection when using memory maps.
+const HAY: &'static [u8] = include_bytes!("./data/sherlock-nul.txt");
+
+// This tests that ripgrep prints a warning message if it finds and prints a
+// match in a binary file before detecting that it is a binary file. The point
+// here is to notify that user that the search of the file is only partially
+// complete.
+//
+// This applies to files that are *implicitly* searched via a recursive
+// directory traversal. In particular, this results in a WARNING message being
+// printed. We make our file "implicit" by doing a recursive search with a glob
+// that matches our file.
+rgtest!(after_match1_implicit, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "Project Gutenberg EBook", "-g", "hay",
+    ]);
+
+    let expected = "\
+hay:1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+WARNING: stopped searching binary file hay after match (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like after_match1_implicit, except we provide a file to search
+// explicitly. This results in identical behavior, but a different message.
+rgtest!(after_match1_explicit, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "Project Gutenberg EBook", "hay",
+    ]);
+
+    let expected = "\
+1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+Binary file matches (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like after_match1_explicit, except we feed our content on stdin.
+rgtest!(after_match1_stdin, |_: Dir, mut cmd: TestCommand| {
+    cmd.args(&[
+        "--no-mmap", "-n", "Project Gutenberg EBook",
+    ]);
+
+    let expected = "\
+1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+Binary file matches (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.pipe(HAY));
+});
+
+// Like after_match1_implicit, but provides the --binary flag, which
+// disables binary filtering. Thus, this matches the behavior of ripgrep as
+// if the file were given explicitly.
+rgtest!(after_match1_implicit_binary, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "--binary", "Project Gutenberg EBook", "-g", "hay",
+    ]);
+
+    let expected = "\
+hay:1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+Binary file hay matches (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like after_match1_implicit, but enables -a/--text, so no binary
+// detection should be performed.
+rgtest!(after_match1_implicit_text, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "--text", "Project Gutenberg EBook", "-g", "hay",
+    ]);
+
+    let expected = "\
+hay:1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like after_match1_implicit_text, but enables -a/--text, so no binary
+// detection should be performed.
+rgtest!(after_match1_explicit_text, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "--text", "Project Gutenberg EBook", "hay",
+    ]);
+
+    let expected = "\
+1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like after_match1_implicit, except this asks ripgrep to print all matching
+// files.
+//
+// This is an interesting corner case that one might consider a bug, however,
+// it's unlikely to be fixed. Namely, ripgrep probably shouldn't print `hay`
+// as a matching file since it is in fact a binary file, and thus should be
+// filtered out by default. However, the --files-with-matches flag will print
+// out the path of a matching file as soon as a match is seen and then stop
+// searching completely. Therefore, the NUL byte is never actually detected.
+//
+// The only way to fix this would be to kill ripgrep's performance in this case
+// and continue searching the entire file for a NUL byte. (Similarly if the
+// --quiet flag is set. See the next test.)
+rgtest!(after_match1_implicit_path, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-l", "Project Gutenberg EBook", "-g", "hay",
+    ]);
+    eqnice!("hay\n", cmd.stdout());
+});
+
+// Like after_match1_implicit_path, except this indicates that a match was
+// found with no other output. (This is the same bug described above, but
+// manifest as an exit code with no output.)
+rgtest!(after_match1_implicit_quiet, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-q", "Project Gutenberg EBook", "-g", "hay",
+    ]);
+    eqnice!("", cmd.stdout());
+});
+
+// This sets up the same test as after_match1_implicit_path, but instead of
+// just printing the matching files, this includes the full count of matches.
+// In this case, we need to search the entire file, so ripgrep correctly
+// detects the binary data and suppresses output.
+rgtest!(after_match1_implicit_count, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-c", "Project Gutenberg EBook", "-g", "hay",
+    ]);
+    cmd.assert_err();
+});
+
+// Like after_match1_implicit_count, except the --binary flag is provided,
+// which makes ripgrep disable binary data filtering even for implicit files.
+rgtest!(after_match1_implicit_count_binary, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-c", "--binary",
+        "Project Gutenberg EBook",
+        "-g", "hay",
+    ]);
+    eqnice!("hay:1\n", cmd.stdout());
+});
+
+// Like after_match1_implicit_count, except the file path is provided
+// explicitly, so binary filtering is disabled and a count is correctly
+// reported.
+rgtest!(after_match1_explicit_count, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-c", "Project Gutenberg EBook", "hay",
+    ]);
+    eqnice!("1\n", cmd.stdout());
+});
+
+// This tests that a match way before the NUL byte is shown, but a match after
+// the NUL byte is not.
+rgtest!(after_match2_implicit, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n",
+        "Project Gutenberg EBook|a medical student",
+        "-g", "hay",
+    ]);
+
+    let expected = "\
+hay:1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+WARNING: stopped searching binary file hay after match (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like after_match2_implicit, but enables -a/--text, so no binary
+// detection should be performed.
+rgtest!(after_match2_implicit_text, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "--text",
+        "Project Gutenberg EBook|a medical student",
+        "-g", "hay",
+    ]);
+
+    let expected = "\
+hay:1:The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+hay:236:\"And yet you say he is not a medical student?\"
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// This tests that ripgrep *silently* quits before finding a match that occurs
+// after a NUL byte.
+rgtest!(before_match1_implicit, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "Heaven", "-g", "hay",
+    ]);
+    cmd.assert_err();
+});
+
+// This tests that ripgrep *does not* silently quit before finding a match that
+// occurs after a NUL byte when a file is explicitly searched.
+rgtest!(before_match1_explicit, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "Heaven", "hay",
+    ]);
+
+    let expected = "\
+Binary file matches (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like before_match1_implicit, but enables the --binary flag, which
+// disables binary filtering. Thus, this matches the behavior of ripgrep as if
+// the file were given explicitly.
+rgtest!(before_match1_implicit_binary, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "--binary", "Heaven", "-g", "hay",
+    ]);
+
+    let expected = "\
+Binary file hay matches (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like before_match1_implicit, but enables -a/--text, so no binary
+// detection should be performed.
+rgtest!(before_match1_implicit_text, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "--text", "Heaven", "-g", "hay",
+    ]);
+
+    let expected = "\
+hay:238:\"No. Heaven knows what the objects of his studies are. But here we
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// This tests that ripgrep *silently* quits before finding a match that occurs
+// before a NUL byte, but within the same buffer as the NUL byte.
+rgtest!(before_match2_implicit, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "a medical student", "-g", "hay",
+    ]);
+    cmd.assert_err();
+});
+
+// This tests that ripgrep *does not* silently quit before finding a match that
+// occurs before a NUL byte, but within the same buffer as the NUL byte. Even
+// though the match occurs before the NUL byte, ripgrep still doesn't print it
+// because it has already scanned ahead to detect the NUL byte. (This matches
+// the behavior of GNU grep.)
+rgtest!(before_match2_explicit, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "a medical student", "hay",
+    ]);
+
+    let expected = "\
+Binary file matches (found \"\\u{0}\" byte around offset 9741)
+";
+    eqnice!(expected, cmd.stdout());
+});
+
+// Like before_match1_implicit, but enables -a/--text, so no binary
+// detection should be performed.
+rgtest!(before_match2_implicit_text, |dir: Dir, mut cmd: TestCommand| {
+    dir.create_bytes("hay", HAY);
+    cmd.args(&[
+        "--no-mmap", "-n", "--text", "a medical student", "-g", "hay",
+    ]);
+
+    let expected = "\
+hay:236:\"And yet you say he is not a medical student?\"
+";
+    eqnice!(expected, cmd.stdout());
+});
diff --git a/tests/data/sherlock-nul.txt b/tests/data/sherlock-nul.txt
new file mode 100644
index 00000000..f1c9db1b
--- /dev/null
+++ b/tests/data/sherlock-nul.txt
@@ -0,0 +1,500 @@
+The Project Gutenberg EBook of A Study In Scarlet, by Arthur Conan Doyle
+
+This eBook is for the use of anyone anywhere at no cost and with
+almost no restrictions whatsoever.  You may copy it, give it away or
+re-use it under the terms of the Project Gutenberg License included
+with this eBook or online at www.gutenberg.org
+
+
+Title: A Study In Scarlet
+
+Author: Arthur Conan Doyle
+
+Posting Date: July 12, 2008 [EBook #244]
+Release Date: April, 1995
+[Last updated: February 17, 2013]
+
+Language: English
+
+
+*** START OF THIS PROJECT GUTENBERG EBOOK A STUDY IN SCARLET ***
+
+
+
+
+Produced by Roger Squires
+
+
+
+
+
+A STUDY IN SCARLET.
+
+By A. Conan Doyle
+
+[1]
+
+
+
+     Original Transcriber's Note: This etext is prepared directly
+     from an 1887 edition, and care has been taken to duplicate the
+     original exactly, including typographical and punctuation
+     vagaries.
+
+     Additions to the text include adding the underscore character to
+     indicate italics, and textual end-notes in square braces.
+
+     Project Gutenberg Editor's Note: In reproofing and moving old PG
+     files such as this to the present PG directory system it is the
+     policy to reformat the text to conform to present PG Standards.
+     In this case however, in consideration of the note above of the
+     original transcriber describing his care to try to duplicate the
+     original 1887 edition as to typography and punctuation vagaries,
+     no changes have been made in this ascii text file. However, in
+     the Latin-1 file and this html file, present standards are
+     followed and the several French and Spanish words have been
+     given their proper accents.
+
+	 Part II, The Country of the Saints, deals much with the Mormon Church.
+
+
+
+
+A STUDY IN SCARLET.
+
+
+
+
+
+PART I.
+
+(_Being a reprint from the reminiscences of_ JOHN H. WATSON, M.D., _late
+of the Army Medical Department._) [2]
+
+
+
+
+CHAPTER I. MR. SHERLOCK HOLMES.
+
+
+IN the year 1878 I took my degree of Doctor of Medicine of the
+University of London, and proceeded to Netley to go through the course
+prescribed for surgeons in the army. Having completed my studies there,
+I was duly attached to the Fifth Northumberland Fusiliers as Assistant
+Surgeon. The regiment was stationed in India at the time, and before
+I could join it, the second Afghan war had broken out. On landing at
+Bombay, I learned that my corps had advanced through the passes, and
+was already deep in the enemy's country. I followed, however, with many
+other officers who were in the same situation as myself, and succeeded
+in reaching Candahar in safety, where I found my regiment, and at once
+entered upon my new duties.
+
+The campaign brought honours and promotion to many, but for me it had
+nothing but misfortune and disaster. I was removed from my brigade and
+attached to the Berkshires, with whom I served at the fatal battle of
+Maiwand. There I was struck on the shoulder by a Jezail bullet, which
+shattered the bone and grazed the subclavian artery. I should have
+fallen into the hands of the murderous Ghazis had it not been for the
+devotion and courage shown by Murray, my orderly, who threw me across a
+pack-horse, and succeeded in bringing me safely to the British lines.
+
+Worn with pain, and weak from the prolonged hardships which I had
+undergone, I was removed, with a great train of wounded sufferers, to
+the base hospital at Peshawar. Here I rallied, and had already improved
+so far as to be able to walk about the wards, and even to bask a little
+upon the verandah, when I was struck down by enteric fever, that curse
+of our Indian possessions. For months my life was despaired of, and
+when at last I came to myself and became convalescent, I was so weak and
+emaciated that a medical board determined that not a day should be lost
+in sending me back to England. I was dispatched, accordingly, in the
+troopship "Orontes," and landed a month later on Portsmouth jetty, with
+my health irretrievably ruined, but with permission from a paternal
+government to spend the next nine months in attempting to improve it.
+
+I had neither kith nor kin in England, and was therefore as free as
+air--or as free as an income of eleven shillings and sixpence a day will
+permit a man to be. Under such circumstances, I naturally gravitated to
+London, that great cesspool into which all the loungers and idlers of
+the Empire are irresistibly drained. There I stayed for some time at
+a private hotel in the Strand, leading a comfortless, meaningless
+existence, and spending such money as I had, considerably more freely
+than I ought. So alarming did the state of my finances become, that
+I soon realized that I must either leave the metropolis and rusticate
+somewhere in the country, or that I must make a complete alteration in
+my style of living. Choosing the latter alternative, I began by making
+up my mind to leave the hotel, and to take up my quarters in some less
+pretentious and less expensive domicile.
+
+On the very day that I had come to this conclusion, I was standing at
+the Criterion Bar, when some one tapped me on the shoulder, and turning
+round I recognized young Stamford, who had been a dresser under me at
+Barts. The sight of a friendly face in the great wilderness of London is
+a pleasant thing indeed to a lonely man. In old days Stamford had never
+been a particular crony of mine, but now I hailed him with enthusiasm,
+and he, in his turn, appeared to be delighted to see me. In the
+exuberance of my joy, I asked him to lunch with me at the Holborn, and
+we started off together in a hansom.
+
+"Whatever have you been doing with yourself, Watson?" he asked in
+undisguised wonder, as we rattled through the crowded London streets.
+"You are as thin as a lath and as brown as a nut."
+
+I gave him a short sketch of my adventures, and had hardly concluded it
+by the time that we reached our destination.
+
+"Poor devil!" he said, commiseratingly, after he had listened to my
+misfortunes. "What are you up to now?"
+
+"Looking for lodgings." [3] I answered. "Trying to solve the problem
+as to whether it is possible to get comfortable rooms at a reasonable
+price."
+
+"That's a strange thing," remarked my companion; "you are the second man
+to-day that has used that expression to me."
+
+"And who was the first?" I asked.
+
+"A fellow who is working at the chemical laboratory up at the hospital.
+He was bemoaning himself this morning because he could not get someone
+to go halves with him in some nice rooms which he had found, and which
+were too much for his purse."
+
+"By Jove!" I cried, "if he really wants someone to share the rooms and
+the expense, I am the very man for him. I should prefer having a partner
+to being alone."
+
+Young Stamford looked rather strangely at me over his wine-glass. "You
+don't know Sherlock Holmes yet," he said; "perhaps you would not care
+for him as a constant companion."
+
+"Why, what is there against him?"
+
+"Oh, I didn't say there was anything against him. He is a little queer
+in his ideas--an enthusiast in some branches of science. As far as I
+know he is a decent fellow enough."
+
+"A medical student, I suppose?" said I.
+
+"No--I have no idea what he intends to go in for. I believe he is well
+up in anatomy, and he is a first-class chemist; but, as far as I know,
+he has never taken out any systematic medical classes. His studies are
+very desultory and eccentric, but he has amassed a lot of out-of-the way
+knowledge which would astonish his professors."
+
+"Did you never ask him what he was going in for?" I asked.
+
+"No; he is not a man that it is easy to draw out, though he can be
+communicative enough when the fancy seizes him."
+
+"I should like to meet him," I said. "If I am to lodge with anyone, I
+should prefer a man of studious and quiet habits. I am not strong
+enough yet to stand much noise or excitement. I had enough of both in
+Afghanistan to last me for the remainder of my natural existence. How
+could I meet this friend of yours?"
+
+"He is sure to be at the laboratory," returned my companion. "He either
+avoids the place for weeks, or else he works there from morning to
+night. If you like, we shall drive round together after luncheon."
+
+"Certainly," I answered, and the conversation drifted away into other
+channels.
+
+As we made our way to the hospital after leaving the Holborn, Stamford
+gave me a few more particulars about the gentleman whom I proposed to
+take as a fellow-lodger.
+
+"You mustn't blame me if you don't get on with him," he said; "I know
+nothing more of him than I have learned from meeting him occasionally in
+the laboratory. You proposed this arrangement, so you must not hold me
+responsible."
+
+"If we don't get on it will be easy to part company," I answered. "It
+seems to me, Stamford," I added, looking hard at my companion, "that you
+have some reason for washing your hands of the matter. Is this fellow's
+temper so formidable, or what is it? Don't be mealy-mouthed about it."
+
+"It is not easy to express the inexpressible," he answered with a laugh.
+"Holmes is a little too scientific for my tastes--it approaches to
+cold-bloodedness. I could imagine his giving a friend a little pinch of
+the latest vegetable alkaloid, not out of malevolence, you understand,
+but simply out of a spirit of inquiry in order to have an accurate idea
+of the effects. To do him justice, I think that he would take it himself
+with the same readiness. He appears to have a passion for definite and
+exact knowledge."
+
+"Very right too."
+
+"Yes, but it may be pushed to excess. When it comes to beating the
+subjects in the dissecting-rooms with a stick, it is certainly taking
+rather a bizarre shape."
+
+"Beating the subjects!"
+
+"Yes, to verify how far bruises may be produced after death. I saw him
+at it with my own eyes."
+
+"And yet you say he is not a medical student?"
+abcdef
+"No. Heaven knows what the objects of his studies are. But here we
+are, and you must form your own impressions about him." As he spoke, we
+turned down a narrow lane and passed through a small side-door, which
+opened into a wing of the great hospital. It was familiar ground to me,
+and I needed no guiding as we ascended the bleak stone staircase and
+made our way down the long corridor with its vista of whitewashed
+wall and dun-coloured doors. Near the further end a low arched passage
+branched away from it and led to the chemical laboratory.
+
+This was a lofty chamber, lined and littered with countless bottles.
+Broad, low tables were scattered about, which bristled with retorts,
+test-tubes, and little Bunsen lamps, with their blue flickering flames.
+There was only one student in the room, who was bending over a distant
+table absorbed in his work. At the sound of our steps he glanced round
+and sprang to his feet with a cry of pleasure. "I've found it! I've
+found it," he shouted to my companion, running towards us with a
+test-tube in his hand. "I have found a re-agent which is precipitated
+by hoemoglobin, [4] and by nothing else." Had he discovered a gold mine,
+greater delight could not have shone upon his features.
+
+"Dr. Watson, Mr. Sherlock Holmes," said Stamford, introducing us.
+
+"How are you?" he said cordially, gripping my hand with a strength
+for which I should hardly have given him credit. "You have been in
+Afghanistan, I perceive."
+
+"How on earth did you know that?" I asked in astonishment.
+
+"Never mind," said he, chuckling to himself. "The question now is about
+hoemoglobin. No doubt you see the significance of this discovery of
+mine?"
+
+"It is interesting, chemically, no doubt," I answered, "but
+practically----"
+
+"Why, man, it is the most practical medico-legal discovery for years.
+Don't you see that it gives us an infallible test for blood stains. Come
+over here now!" He seized me by the coat-sleeve in his eagerness, and
+drew me over to the table at which he had been working. "Let us have
+some fresh blood," he said, digging a long bodkin into his finger, and
+drawing off the resulting drop of blood in a chemical pipette. "Now, I
+add this small quantity of blood to a litre of water. You perceive that
+the resulting mixture has the appearance of pure water. The proportion
+of blood cannot be more than one in a million. I have no doubt, however,
+that we shall be able to obtain the characteristic reaction." As he
+spoke, he threw into the vessel a few white crystals, and then added
+some drops of a transparent fluid. In an instant the contents assumed a
+dull mahogany colour, and a brownish dust was precipitated to the bottom
+of the glass jar.
+
+"Ha! ha!" he cried, clapping his hands, and looking as delighted as a
+child with a new toy. "What do you think of that?"
+
+"It seems to be a very delicate test," I remarked.
+
+"Beautiful! beautiful! The old Guiacum test was very clumsy and
+uncertain. So is the microscopic examination for blood corpuscles. The
+latter is valueless if the stains are a few hours old. Now, this appears
+to act as well whether the blood is old or new. Had this test been
+invented, there are hundreds of men now walking the earth who would long
+ago have paid the penalty of their crimes."
+
+"Indeed!" I murmured.
+
+"Criminal cases are continually hinging upon that one point. A man is
+suspected of a crime months perhaps after it has been committed. His
+linen or clothes are examined, and brownish stains discovered upon them.
+Are they blood stains, or mud stains, or rust stains, or fruit stains,
+or what are they? That is a question which has puzzled many an expert,
+and why? Because there was no reliable test. Now we have the Sherlock
+Holmes' test, and there will no longer be any difficulty."
+
+His eyes fairly glittered as he spoke, and he put his hand over his
+heart and bowed as if to some applauding crowd conjured up by his
+imagination.
+
+"You are to be congratulated," I remarked, considerably surprised at his
+enthusiasm.
+
+"There was the case of Von Bischoff at Frankfort last year. He would
+certainly have been hung had this test been in existence. Then there was
+Mason of Bradford, and the notorious Muller, and Lefevre of Montpellier,
+and Samson of New Orleans. I could name a score of cases in which it
+would have been decisive."
+
+"You seem to be a walking calendar of crime," said Stamford with a
+laugh. "You might start a paper on those lines. Call it the 'Police News
+of the Past.'"
+
+"Very interesting reading it might be made, too," remarked Sherlock
+Holmes, sticking a small piece of plaster over the prick on his finger.
+"I have to be careful," he continued, turning to me with a smile, "for I
+dabble with poisons a good deal." He held out his hand as he spoke, and
+I noticed that it was all mottled over with similar pieces of plaster,
+and discoloured with strong acids.
+
+"We came here on business," said Stamford, sitting down on a high
+three-legged stool, and pushing another one in my direction with
+his foot. "My friend here wants to take diggings, and as you were
+complaining that you could get no one to go halves with you, I thought
+that I had better bring you together."
+
+Sherlock Holmes seemed delighted at the idea of sharing his rooms with
+me. "I have my eye on a suite in Baker Street," he said, "which would
+suit us down to the ground. You don't mind the smell of strong tobacco,
+I hope?"
+
+"I always smoke 'ship's' myself," I answered.
+
+"That's good enough. I generally have chemicals about, and occasionally
+do experiments. Would that annoy you?"
+
+"By no means."
+
+"Let me see--what are my other shortcomings. I get in the dumps at
+times, and don't open my mouth for days on end. You must not think I am
+sulky when I do that. Just let me alone, and I'll soon be right. What
+have you to confess now? It's just as well for two fellows to know the
+worst of one another before they begin to live together."
+
+I laughed at this cross-examination. "I keep a bull pup," I said, "and
+I object to rows because my nerves are shaken, and I get up at all sorts
+of ungodly hours, and I am extremely lazy. I have another set of vices
+when I'm well, but those are the principal ones at present."
+
+"Do you include violin-playing in your category of rows?" he asked,
+anxiously.
+
+"It depends on the player," I answered. "A well-played violin is a treat
+for the gods--a badly-played one----"
+
+"Oh, that's all right," he cried, with a merry laugh. "I think we may
+consider the thing as settled--that is, if the rooms are agreeable to
+you."
+
+"When shall we see them?"
+
+"Call for me here at noon to-morrow, and we'll go together and settle
+everything," he answered.
+
+"All right--noon exactly," said I, shaking his hand.
+
+We left him working among his chemicals, and we walked together towards
+my hotel.
+
+"By the way," I asked suddenly, stopping and turning upon Stamford, "how
+the deuce did he know that I had come from Afghanistan?"
+
+My companion smiled an enigmatical smile. "That's just his little
+peculiarity," he said. "A good many people have wanted to know how he
+finds things out."
+
+"Oh! a mystery is it?" I cried, rubbing my hands. "This is very piquant.
+I am much obliged to you for bringing us together. 'The proper study of
+mankind is man,' you know."
+
+"You must study him, then," Stamford said, as he bade me good-bye.
+"You'll find him a knotty problem, though. I'll wager he learns more
+about you than you about him. Good-bye."
+
+"Good-bye," I answered, and strolled on to my hotel, considerably
+interested in my new acquaintance.
+
+
+
+
+CHAPTER II. THE SCIENCE OF DEDUCTION.
+
+
+WE met next day as he had arranged, and inspected the rooms at No. 221B,
+[5] Baker Street, of which he had spoken at our meeting. They
+consisted of a couple of comfortable bed-rooms and a single large
+airy sitting-room, cheerfully furnished, and illuminated by two broad
+windows. So desirable in every way were the apartments, and so moderate
+did the terms seem when divided between us, that the bargain was
+concluded upon the spot, and we at once entered into possession.
+That very evening I moved my things round from the hotel, and on the
+following morning Sherlock Holmes followed me with several boxes and
+portmanteaus. For a day or two we were busily employed in unpacking and
+laying out our property to the best advantage. That done, we
+gradually began to settle down and to accommodate ourselves to our new
+surroundings.
+
+Holmes was certainly not a difficult man to live with. He was quiet
+in his ways, and his habits were regular. It was rare for him to be
+up after ten at night, and he had invariably breakfasted and gone out
+before I rose in the morning. Sometimes he spent his day at the chemical
+laboratory, sometimes in the dissecting-rooms, and occasionally in long
+walks, which appeared to take him into the lowest portions of the City.
+Nothing could exceed his energy when the working fit was upon him; but
+now and again a reaction would seize him, and for days on end he would
+lie upon the sofa in the sitting-room, hardly uttering a word or moving
+a muscle from morning to night. On these occasions I have noticed such
+a dreamy, vacant expression in his eyes, that I might have suspected him
+of being addicted to the use of some narcotic, had not the temperance
+and cleanliness of his whole life forbidden such a notion.
+
+As the weeks went by, my interest in him and my curiosity as to his
+aims in life, gradually deepened and increased. His very person and
+appearance were such as to strike the attention of the most casual
+observer. In height he was rather over six feet, and so excessively
+lean that he seemed to be considerably taller. His eyes were sharp and
+piercing, save during those intervals of torpor to which I have alluded;
+and his thin, hawk-like nose gave his whole expression an air of
+alertness and decision. His chin, too, had the prominence and squareness
+which mark the man of determination. His hands were invariably
+blotted with ink and stained with chemicals, yet he was possessed of
+extraordinary delicacy of touch, as I frequently had occasion to observe
+when I watched him manipulating his fragile philosophical instruments.
+
+The reader may set me down as a hopeless busybody, when I confess how
+much this man stimulated my curiosity, and how often I endeavoured
+to break through the reticence which he showed on all that concerned
+himself. Before pronouncing judgment, however, be it remembered, how
+objectless was my life, and how little there was to engage my attention.
+My health forbade me from venturing out unless the weather was
+exceptionally genial, and I had no friends who would call upon me and
+break the monotony of my daily existence. Under these circumstances, I
+eagerly hailed the little mystery which hung around my companion, and
+spent much of my time in endeavouring to unravel it.
+
+He was not studying medicine. He had himself, in reply to a question,
+confirmed Stamford's opinion upon that point. Neither did he appear to
+have pursued any course of reading which might fit him for a degree in
+science or any other recognized portal which would give him an entrance
+into the learned world. Yet his zeal for certain studies was remarkable,
+and within eccentric limits his knowledge was so extraordinarily ample
+and minute that his observations have fairly astounded me. Surely no man
+would work so hard or attain such precise information unless he had some
+definite end in view. Desultory readers are seldom remarkable for the
+exactness of their learning. No man burdens his mind with small matters
+unless he has some very good reason for doing so.
+
+His ignorance was as remarkable as his knowledge. Of contemporary
+literature, philosophy and politics he appeared to know next to nothing.
+Upon my quoting Thomas Carlyle, he inquired in the naivest way who he
+might be and what he had done. My surprise reached a climax, however,
+when I found incidentally that he was ignorant of the Copernican Theory
+and of the composition of the Solar System. That any civilized human
+being in this nineteenth century should not be aware that the earth
+travelled round the sun appeared to be to me such an extraordinary fact
+that I could hardly realize it.
+
+"You appear to be astonished," he said, smiling at my expression of
+surprise. "Now that I do know it I shall do my best to forget it."
+
+"To forget it!"
+
+"You see," he explained, "I consider that a man's brain originally is
+like a little empty attic, and you have to stock it with such furniture
+as you choose. A fool takes in all the lumber of every sort that he
+comes across, so that the knowledge which might be useful to him gets
+crowded out, or at