Latest posts

Forum Statistics

Threads
28,136
Posts
553,145
Members
28,724
Latest Member
TeeJay
What's New?
mugzy

mugzy

MuscleHead
Aug 11, 2010
4,876
1,806
Memex crawls content ignored by commercial search engines and uncovers patterns in online data that can aid law enforcement

The government agency that brought us the Internet has now developed a powerful new search engine that is shedding light on the contents of the so-called deep Web.
The Defense Advanced Research Projects Agency (DARPA) began work on the Memex Deep Web Search Engine a year ago, and this week unveiled its tools to Scientific American and "60 Minutes."

Memex, which is being developed by 17 different contractor teams, aims to build a better map of Internet content and uncover patterns in online data that could help law enforcement officers and others. While early trials have focused on mapping the movements of human traffickers, the technology could one day be applied to investigative efforts such as counterterrorism, missing persons, disease response, and disaster relief.

Dan Kaufman, director of the information innovation office at DARPA, says Memex is all about making the unseen seen. "The Internet is much, much bigger than people think," DARPA program manager Chris White told "60 Minutes." "By some estimates Google, Microsoft Bing, and Yahoo only give us access to around 5 percent of the content on the Web."

Google and Bing produce results based on popularity and ranking, but Memex searches content typically ignored by commercial search engines, such as unstructured data, unlinked content, temporary pages that are removed before commercial search engines can crawl them, and chat forums. Regular search engines ignore this deep Web data because Web advertisers -- where browser companies make their money -- have no interest in it.

Memex also automates the mechanism of crawling the dark, or anonymous, Web where criminals conduct business. These hidden services pages, accessible only through the TOR anonymizing browser, typically operate under the radar of law enforcement selling illicit drugs and other contraband. Where it was once thought that dark Web activity consisted of 1,000 or so pages, White told Scientific American that there could be between 30,000 and 40,000 dark Web pages.

Until now it was hard to look at these sites in any systemic way. But Memex -- which Manhattan DA Cyrus Vance Jr. calls "Google search on steroids" -- not only indexes their content but analyzes it to uncover hidden relationships that could be useful to law enforcement.

DARPA's search tools were introduced to select law enforcement agencies last year, including Manhattan's new Human Trafficking Response Unit. Memex is now used in every human trafficking case it pursues and has played a role in generating at least 20 sex trafficking investigations. The supercharged Web crawler can identify relationships among different pieces of data and produces data maps that help investigators detect patterns.

In a demo for "60 Minutes," White showed how Memex is able to track the movement of drug traffickers based on data related to online advertisements for drugs or sex. "Sometimes it's a function of IP address, but sometimes it's a function of a phone number or address in the ad or the geolocation of a device that posted the ad," White said. "There are sometimes other artifacts that contribute to location."

White emphasized that Memex does not resort to hacking in order to retrieve information. "If something is password protected, it is not public content and Memex does not search it," he told Scientific American. "We didn't want to cloud this work unnecessarily by dragging in the specter of snooping and surveillance" -- a touchy subject after Edward Snowden's NSA revelations.

Memex got its name (a combination of "memory" and "index") and inspiration from a hypothetical device described by Vannevar Bush in 1945 that presaged the invention of PCs, the Internet, and other major IT advances of the next 70 years. Now DARPA and Memex seem set to bring us one step closer to Philip Dick's futuristic police department depicted in "Minority Report."

A new round of testing, set to begin in a few weeks, will include federal and district prosecutors, regional and national law enforcement, and multiple NGOs. According to the Scientific American report, it aims to "test new image search capabilities that can analyze photos even when portions that might aid investigators -- including traffickers' faces or a television screen in the background -- are obfuscated."

By inventing better ways of interacting with and presenting information gathered from a larger pool of sources, "we want to improve search for everybody. Ease of use for nonprogrammers is essential," White said.


Code:
http://www.infoworld.com/article/2883793/government/darpa-search-engine-casts-light-on-dark-web.html
 
monsoon

monsoon

Senior Bacon VIP
Nov 1, 2010
5,254
2,104
Not really sure how this will aid in disaster relief?
 
PillarofBalance

PillarofBalance

Strength Pimp
Feb 27, 2011
17,066
4,641
Well I certainly have some questions on this thing...
 
Lizard King

Lizard King

Administrator
Staff Member
Sep 9, 2010
14,667
8,310
which Manhattan DA Cyrus Vance Jr. calls "Google search on steroids"

Yep, cause these are the worst things sold on the dark web, where do they find these idiots to even quote?
 
woodswise

woodswise

TID Board Of Directors
Apr 29, 2012
4,348
1,358
"Google Search on Steroids"

Ironic how they use the Steroids analogy to describe something they view as powerful and good, while they are also trying to put a stop to the use of steroids . . .
 
IronCore

IronCore

Bigger Than MAYO - VIP
Sep 9, 2010
4,321
1,539
sounds like there needs to be a password on the forum splash screen.. no more public browsing.
 
Who is viewing this thread?

There are currently 0 members watching this topic

Top