I have no data yet. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
| osintbuddy-plugins | osintbuddy.com | forum.osintbuddy.com | osintbuddy-core-plugins |
Introducing OSINTBuddy
Welcome to the OSINTBuddy project where you can connect, combine, and get insight from unstructured and public data as results that can be explored step-by-step. An easy-to-use plugin system allows any Python developer to quickly integrate new data sources.
🚧 Work in progress 🚧
⚠️ Searching for contributors, if you want to help, please contact me on the forum or on the discord ⚠️
Table of Contents
What is osintbuddy.com?
Not much, yet.
But here's the concept:
An almost incomprehensible amount of data is created every day. And each year, figures are growing at an ever-increasing rate. These data sources can be divided up into six different categories of information flow:
- Public government data
- Media (newspapers, magazines, radio)
- Internet (blogs, discussion groups, citizen media, etc)
- Professional and academic publications (budgets, hearings, telephone directories, websites, etc)
- Commercial data (commercial imagery, financial assessments, databases, etc)
- Grey literature (technical reports, preprints, patents, business documents, etc)
Some of the main hindrances to practical OSINT is the volume of information it has to deal with (information explosion) and the issue of low quality data. With the majority of this data being unstructured, there's many challenges to analyzing it and producing actionable intelligence since most data analytics databases are designed for structured data.
We've decided to do something about it. The rapid developments in technologies such as AI and big data analytics have opened new horizons for OSINT which weren't previously available. We want our information quick and to the point, that's why we're building an open-source OSINT tool that's free. Free to use, free to modify, free to do with as you wish, and built with plain old web technologies anyone can learn. But this isn't really the project. This is a free new community. A community for OSINT enthusiast around the world and we need your help to design it, to program it, and to build it. We want to hear your suggestions, your ideas, and we're going to build it right in front of your eyes. The notion of a “needle in a haystack” is taken to the extreme on the internet. Let's build a magnet.
Key Features
- Simplified data fetching from multiple sources.
- Visual representation of fetched data for easy understanding.
- A development platform that is open for contributions.
- Plugin-based system for transforming data, check out the osintbuddy package.
Getting Started
To start using OSINTBuddy, follow these simple installation steps:
Installation
- Clone the repo and submodules
git clone --depth=1 --recurse-submodules https://github.com/jerlendds/osintbuddy.git cd osintbuddy - Install Docker
- Start the stack with Docker:
# ./launcher # display usage information. ./launcher bootstrap # first time? build images; setup ui and backend dependencies; ./launcher start # start the osintbuddy stack.- Note: the stack will take a few minutes to startup while Solr and ScyllaDB configure themselves for JanusGraph. If you try to connect before all the databases are ready you will encounter errors.
- URLs
- Frontend: http://localhost:3000
- Casdoor: http://localhost:8080
- Backend: http://localhost:8000/api
- Documentation: http://localhost:8000/docs
- Access OSINTBuddy through the URLs provided for the frontend, backend, and documentation.
↑Roadmap
See our trello board
for a list of our upcoming features.
See the forum: forum.osintbuddy.com, for a list of our upcoming features, potential ideas, and more (Note: We're currently in the process of setting up the forum and moving trello issues over. We're not sure when this will be completed. For the time being you can count on dev log updates being on the forum from now on).
See the open issues for a list of requested features (and known issues).
↑Progress Notes
- Website node
- Google search node
- Google cache search node
- Google dorks
- DNS node
- URL node
- IP node
- CSE node
- Username node
- Added a plugin system
- Updated plugin system to be easier to work with
- Implemented automatic saving for project graphs
- Added
obandlauncherCLI tools - Added Casdoor for auth
- Improved UI/UX
[] Add wizards for plugin create and custom .env setup
[] Rework plugin system to its own service
[] Add playwright and start reworking existing selenium based plugins
For this update you'll have to rebuild your docker containers (./launcher build)! After a long wait you'll be rewarded with a new UX, an updated plugin system now running as its own service, Casdoor managed auth for signing in/up, a new CLI tool, ob, for managing plugin development and the plugin server, and another CLI ./launcher for managing the OSINTBuddy stack.
↑Contributing
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/add-core-feature) - Commit your Changes (
git commit -m 'feat: add core feature to osintbuddy app') - Push to the Branch (
git push origin feature/add-core-feature) - Open a Pull Request
↑License
Licensed under the GNU Affero General Public License v3.0.
Copyright 2023 jerlendds. GNU AGPL v3.
Note: the OSINTBuddy PyPi package is MIT licensed.
Patched aiogremlin library: jerlendds/gremlinpy
↑Contact
Open an issue if you need to get in touch with me or send an email to jerlendds@osintbuddy.com.
↑Sponsor OSINTBuddy
Help us keep the OSINTBuddy project free and maintained forever. Sponsor the OSINTBuddy project for unique benefits. Learn more on the OSINTBuddy OpenCollective project page