I have came across one scenario where I have to send .csv file containing data to Tableau for analytics.
The producer of the data is another external application and I am using my MFT (managed file transfer) to get the data using SFTP.
I was wondering if Tableau server or desktop (preferably Tableau Server) has any file storage medium where I can place the .csv file and that can be use by Tableau to produce reports and analytics.
I know that Tableau has few connectors for database that can fetch the records from database but I dont want in that way.
I want to rather place the file in file server location from where Tableau can pick that data as input feed. If yes this is possible then does tableau server has any inbuilt file server location which can be used to place the file in dedicated location.
I am not a Tableau expert, so please bear with my little knowledge on explaining Tableau things.
Thanks in advance.
Related
I store large files (50-500MB) in database. Once loaded by the application, it doesn't need the whole file in memory. How do i fetch table row (or specifically the installer from the row) directly into the memory while avoiding loading the entire file into ram (So a sort of a buffered download into file)?
I haven't found a solution that avoid loading the file so far. Instead i forward requests to flask server that loads the entire file, and then allows the application instance to download it into a file. However this doesn't seem like a very good solution.
You are probably looking for FILESTREAM (SQL Server):
FILESTREAM enables SQL Server-based applications to store unstructured data, such as documents and images, on the file system. Applications can leverage the rich streaming APIs and performance of the file system and at the same time maintain transactional consistency between the unstructured data and corresponding structured data.
It is interesting because on SQL Server (for Windows) it can stream file data to Windows clients without having to load their entirety into the memory of the SQL Server:
The Win32 streaming support works in the context of a SQL Server transaction. Within a transaction, you can use FILESTREAM functions to obtain a logical UNC file system path of a file. You then use the OpenSqlFilestream API to obtain a file handle. This handle can then be used by Win32 file streaming interfaces, such as ReadFile() and WriteFile(), to access and update the file by way of the file system.
Do note that at this time it is not supported on SQL Server 2017 for Linux.
The overall goal is to have data from an automated daily Cognos report stored in a database so that I am able to report not only on that day but also historical data if I so choose. My general thought is that if I can find a way to automatically add the new daily data to an existing Excel file, I can then use that as my data source and create a dashboard in Tableau. However, I don't have any programming experience, so I'm floundering here.
I'm committed to using Tableau, but I chose Excel only because I'm more familiar with that program than others, along with the fact that an Excel output file is an option in Cognos. If you have better ideas, please don't hesitate to suggest them along with why you believe it's a better idea.
Update: I'm still jumping through hoops to try to get read-only access to the backend database to make this process a lot more efficient, but in the meantime I've moved forward with the long method utilizing Cognos.
I was able to leverage a coworker to create a system file folder to automatically save the Cognos reports to, and then I scheduled a job to run the reports I need. Each of those now saves into a folder in a shared network drive (so my entire team has access to the files), and I wrote a series of macros to append the data each day from those feeder files in the shared drive to a Master File. Now all that's left is to create a Tableau dashboard using the Master File as the data source and I'll have what I need.
Thanks for all your help!
I'm posting this an an answer because, it's just too much to leave as a comment.
What you need are 3 things.
Figure out how to have COGNOS run your report and download your Excel file.
Use Visual Studio with BIDS (which is the suite of SQL analysis, reporting, and integration services) to automate all the stuff you need to do to append your Excel files, etc... Then you can use the same tools to import that data to your SQL server.
In fact, if all you're doing is trying to get this data into SQL, you can skip the Append Excel part, and just append the data directly to your SQL table.
Once your package is built, you can save it as an automated job on your SQL server to run whenever you wish.
Tableau can use your SQL server as a data source. Once you have that updated, you can run your reports.
I've been searching around and haven't found anything on my scenario that I understand:
I have a list of all of the Oracle databases and corresponding servers that my company owns (about 80 servers 150 databases). I am trying to figure out which one a specific file is being downloaded from (from a webpage).
I am mechanical engineer, not in software so if you could eli5 that would be very helpful.
Specifically I need the SID name, but figuring out the server name
would also be helpful.
Your question is kind of tricky here. if your downloading the file from web application(I assuming it is a Java webapp), oracle database could act as either the data store or a report server that can generate the oracle reports directly
In the first case, you need to find out if what kind of file you are downloading?
is it a PDF? is it a excel file? or just text file or anything? the best idea is to check out the file link and then decide what software generating this file. it could be any software in back end to generate the file like, POI(for generating excel file), or even a direct file link, but not oracle at all.
Also, In this case, the file is usually generated at backend by server-let. You need ask the developer which report or file generating engine they are employing. and if oracle database is also being used, it is usually providing the data fro that report or file engine.
In the second case, you can just check out the the URL and give it to the webmaster asking them which oracle server it is using. it is usually configured in the web server.
I am writing an asp.net web application that stores APPLICANTS data in a SQL Server database.
Applicant might post name, address, telephone and a file.
The file might be of any extension including .docx for resume, 'jpg, .pdf for photos.
or even an Excel file.
Is it possible to store all these file extension on my database?
Or will that be lengthy?
Please help
Good question! Personally I would use FILESTREAM in your case and here's why
In SQL Server, BLOBs can be standard varbinary(max) data that stores
the data in tables, or FILESTREAM varbinary(max) objects that store
the data in the file system. The size and use of the data determines
whether you should use database storage or file system storage. If the
following conditions are true, you should consider using FILESTREAM:
Objects that are being stored are, on average, larger than 1 MB.
Fast read access is important.
You are developing applications that use a middle tier for application logic.
For smaller objects, storing varbinary(max) BLOBs
in the database often provides better streaming performance.
You can read up on FILESTREAM here.
Also consider using it in conjunction with FILETABLE.
Finally, here's a .net C# example on how to read from FILESTREAM column.
Please note, FILESTREAM is available in SQL Server starting from 2008 version.
Hope it helps!
I have a standard WinForms application that connects to a SQL Server. The application allows users to upload documents which are currently stored in the database, in a table using an image column.
I need to change this approach so the documents are stored as files and a link to the file is stored in the database table.
Using the current approach - when the user uploads a document they are shielded from how this is stored, as they have a connection to the database they do not need to know anything about where the files are stored, no special directory permissions etc are required. If I set up a network share for the documents I want to avoid any IT issues such as the users having to have access to this directory to upload to or access existing documents.
What are the options available to do this? I thought of having a temporary database where the documents are uploaded to in the same way as the current approach and then a process running on the server to save these to the file store. This database could then be deleted and recreated to reclaim any space. Are there any better approaches?
ADDITIONAL INFO: There is no web server element to my application so I do not think a WCF service is possible
Is there a reason why you want to get the files out of the database in the first place?
How about still saving them in SQL Server, but using a FILESTREAM column instead of IMAGE?
Quote from the link:
FILESTREAM enables SQL Server-based applications to store unstructured
data, such as documents and images, on the file system. Applications
can leverage the rich streaming APIs and performance of the file
system and at the same time maintain transactional consistency between
the unstructured data and corresponding structured data.
FILESTREAM integrates the SQL Server Database Engine with an NTFS file
system by storing varbinary(max) binary large object (BLOB) data as
files on the file system. Transact-SQL statements can insert, update,
query, search, and back up FILESTREAM data. Win32 file system
interfaces provide streaming access to the data.
FILESTREAM uses the NT system cache for caching file data. This helps
reduce any effect that FILESTREAM data might have on Database Engine
performance. The SQL Server buffer pool is not used; therefore, this
memory is available for query processing.
So you would get the best out of both worlds:
The files would be stored as files on the hard disk (probabl faster compared to storing them in the database), but you don't have to care about file shares, permissions etc.
Note that you need at least SQL Server 2008 to use FILESTREAM.
I can tell you how I implemented this task. I wrote a WCF service which is used to send archived files. So, if I were you, I would create such a service which should be able to save files and send them back. This is easy and you also must be sure that the user under which context the WCF service works has permission to read write files.
You could just have your application pass the object to a procedure (CLR maybe) in the database which then writes the data out to the location of your choosing without storing the file contents. That way you still have a layer of abstraction between the file store and the application but you don't need to have a process which cleans up after you.
Alternatively a WCF/web service could be created which the application connects to. A web method could be used to accept the file contents and write them to the correct place, it could return the path to the file or some file identifier.