I have no data yet. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
| osintbuddy-plugins | osintbuddy.com | forum.osintbuddy.com | osintbuddy-core-plugins |
Introducing OSINTBuddy
Welcome to the OSINTBuddy project where you can connect, combine, and get insight from unstructured and public data as results that can be explored step-by-step. An easy-to-use plugin system allows any Python developer to quickly integrate new data sources.
🚧 Work in progress 🚧
⚠️ Searching for contributors, if you want to help, please contact me on the forum or on the discord ⚠️
Table of Contents
What is osintbuddy.com?
Not much, yet.
But here's the concept:
An almost incomprehensible amount of data is created every day. And each year, figures are growing at an ever-increasing rate. These data sources can be divided up into six different categories of information flow:
- Public government data
- Media (newspapers, magazines, radio)
- Internet (blogs, discussion groups, citizen media, etc)
- Professional and academic publications (budgets, hearings, telephone directories, websites, etc)
- Commercial data (commercial imagery, financial assessments, databases, etc)
- Grey literature (technical reports, preprints, patents, business documents, etc)
OSINT is a method of working with, assessing and ranking information — We are living in the information age and the volume of information OSINT research has to deal with (information explosion) and the issue of low quality data leaves researchers drowning. The majority of this data is unstructured introducing many challenges to analyzing it and producing actionable intelligence since most data analytics databases are designed for structured data. We are surrounded by information and nearly none of it is useful.
We've decided to do something about it. The rapid developments in technologies such as AI and big data analytics have opened new horizons for OSINT which weren't previously available. We want to put all of that information at your fingertips. We want actionable intelligence quickly and to the point, that's why we're building an open-source OSINT tool that's free. Free to use, free to modify, free to do with as you wish, and built with plain old web technologies anyone can learn. But this isn't really the project. This is a free new community. A community for OSINT enthusiasts around the world and we need your help to design it, to program it, and to build it. We want to hear your suggestions, your ideas, and we're going to build it right in front of your eyes. The notion of a “needle in a haystack” is taken to the extreme on the internet. Let's build a magnet.
Key Alpha Features
- Simplified data fetching from custom sources using Python plugins.
- Visual representation of fetched data for easy understanding.
- A development platform that is open for contributions.
- Plugin-based system for transforming data, check out the osintbuddy package.
- Works most of the time :)
The future
At the core of this project lies a far more ambitious vision than the mere creation of a data aggregation tool. What we aspire to start is the birth of a knowledge-driven community, passionately dedicated to the development of an evolving intelligence tool - a system and platform designed for the discovery, interaction, and visualization of information to derive actionable insights. We imagine a system that not only learns from and evolves with human intellectual capabilities but also augments the same. We dream of a future where the vast expanse of information online, instead of being insurmountable, becomes a well-organized catalogue, where finding the "needle" is as straightforward as requesting it. Here are some of our ideas for the future in no particular order:
Collective intelligence through collaborative workspaces
- We want to create a platform that promotes collective intelligence. Consider currently popular collaborative workspaces such as Notion, Logseq, or Obsidian integrated seamlessly into our data workspace. A system where each individual user contributes to the pool of knowledge, where insights from your workspaces can be published and engaged with by a global community. Where every connection made, every data source integrated, every plugin created and shared by members of our community contributes to advancing your insights.
Intuitive search and discovery tools
- We want you to be able to interact with our algorithms, search for particular data, apply advanced query filters, data layouts, and more so you can uncover insights that matter via an easy-to-navigate web-based interface.
Real-time monitoring
- This functionality aims to provide real-time updates, alerting users to new relevant data, changes in trends, and emerging patterns in their field of interest, offering insights as they unfold.
AI, swarm intelligence, and evolutionary algorithms
- Inspired by nature, where simple creatures achieve complex tasks through collective decision making, our tool will mimic these principles where it can. Integrating AI, swarm intelligence, and evolutionary algorithms could let us create a tool that can continually evolve and optimize its operations over time. Imagine an intelligence system that fluidly adapts to the ever-changing landscapes of the internet.
Getting Started
To start using OSINTBuddy, follow these simple installation steps:
Installation
-
Clone the repo and submodules
git clone --recurse-submodules https://github.com/jerlendds/osintbuddy.git cd osintbuddy # using ssh? # git clone --recurse-submodules git@github.com:jerlendds/osintbuddy.git -
Install Docker & Compose
-
Start the stack with Docker:
# ./launcher # display usage information. # ./launcher bootstrap # first time working on osintbuddy? build images and # setup development environments for the frontend and backend. ./launcher start # start the osintbuddy stack.- Note: the stack will take a few minutes to startup while Solr and ScyllaDB configure themselves for JanusGraph. If you try to connect before all the databases are ready you will encounter errors.
- URLs
- Frontend: http://localhost:3000
- Casdoor: http://localhost:45910
- Backend: http://localhost:48997/api
- Documentation: http://localhost:48997/docs
- Access OSINTBuddy through the URLs provided for the frontend, backend, and documentation.
↑Roadmap
See the trello board for a list of our upcoming features.
See the forum: forum.osintbuddy.com, for a list of our upcoming features, to contribute feedback, ideas, and more (Note: We're currently in the process of setting up the forum and moving trello issues over. We're not sure when this will be completed. For the time being you can count on dev log updates being on the forum or discord from now on).
See the open issues for a list of requested features (and known issues).
↑Contributing
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated. But if you'd like to make a significant change to this project or the osintbuddy-plugins project, please first create an issue or open a contributor post on the forum to get feedback before spending too much time. We don't want you to invest your time on changes we are already working on.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/add-core-feature) - Commit your Changes (
git commit -m 'feat: add core feature to osintbuddy app') - Push to the Branch (
git push origin feature/add-core-feature) - Open a Pull Request
↑License
We are using the GNU Affero General Public License v3.0
In the early eighties, RMS realised that software was transformed from "a way to use a machine" to a product or a commodity. He foresaw that this would put an end to collective intelligence and to knowledge sharing. He also foresaw that if we were not the master of our software, we would quickly become the slave of the machines controlled by soulless corporations. For the last 30 years, businesses and proponents of Open Source, including Linus Torvalds, have been decrying the GPL because of the essential right of "doing business" aka "privatising the common". Microsoft, through Github, Google and Apple pushed for MIT/BSD licensed software as the open source standard.
Our freedoms were taken away by proprietary software which is mostly coded by ourselves. For free. We spent our free time developing, debugging, testing software before handing them to corporations that we rever, hoping to maybe get a job offer or a small sponsorship from them. Without Non-copyleft Open Source, there would be no proprietary MacOS, OSX nor Android. There would be no Facebook, no Amazon. We created all the components...
The four freedoms and one obligation of free software
- The right to use the software at your own discretion
- The right to study the software
- The right to modify the software
- The right to redistribute the software, including with modifications
- The obligation to keep those four rights, effectively keeping the software in the commons.
We need to realise that any software without that last obligation will, sooner or later, become an oppression tool against ourselves. And that maintaining the commons is not only about software. It’s about everything we are as a society and everything we are losing against individual greed. Ultimately, our planet is our only common resource. We should defend it from becoming a commodity.
Note: the OSINTBuddy PyPi package is MIT licensed.
Patched aiogremlin library: jerlendds/gremlinpy
↑Contact
Open an issue if you need to get in touch with me or send an email to jerlendds@osintbuddy.com.
↑Sponsor OSINTBuddy
Help us keep the project free and maintained. Sponsor the OSINTBuddy project for unique benefits. Learn more on the OSINTBuddy OpenCollective