You've undoubtedly posted data for forms many times, and every time you posted data you probably used an Internet browser. What you might not know: You can call a CGI program yourself without loading your browser, typing information, and clicking on a button. The benefit of using a programmatic interface instead of a browser interface is that you can write a program to do the work, instead of having users open a browser and perform all the steps needed when posting form information over the Web.
For example, you can create a program with the ability to check for the latest version of a digital certificate and notify the user of the result. Calling a CGI program over the Internet, passing it information, and retrieving a response saves time and is just as effective from within a Visual Basic program. In this article, I'll demystify the process of connecting to a server-hosted CGI program, using a client program written in VB. You'll create a module you can use immediately to connect to any program running on any server you have access to.
Posting form data when using a browser usually follows this pattern: You go to a Web site, fill out a form, and press a Submit button. As you wait, if you watch the URL navigation window in your browser, you can see that your browser is probably connecting to a form-handling CGI program or Active Server Page (ASP), which works just like a form-handling CGI program.
My company recently needed to give users the ability to post data to a form, but we didn't want to make our users connect to the Internet, fire up a browser, and type in a lot of information. We realized we already had all the tools we needed: Windows, Internet Explorer's Internet API, and VB. We used WinInet.dll (for HTTP and TCP/IP) and standard VB code for both the client application and the server program. On the server, we used CGI (written in VB) instead of a WebClass or an ASP page because a "real" program gave us complete control over the entire process. The end result is a powerful and flexible system for communicating with your server, with no custom controls, and no expensive software or hardware required (see Figure 1).
Using the Internet for your own diabolical purposes is smart. The Internet is already there, it works (for the most part), and your users already have access to it. The days of building a custom data network infrastructure are over. The beauty of the Internet is that it puts 99 percent of the infrastructure burden squarely on the shoulders of your customers: They get an ISP, they purchase a modem (or whatever they use to access the Internet), and they set up their own PC. You only have to make sure Internet access is available, and you can ride on top of the existing infrastructure.
IE Without a Browser
Prior to IE 4.x, Microsoft distributed a system component called Wintdist.exe, a part of the ActiveX SDK. Wintdist contains the core system files required for Internet communication. It doesn't include a browser, just three critical files: WinInet.dll, InLoader.dll, and Schannel.dll. All the APIs contained in WinInet.dll, InLoader.dll, and Schannel.dll are dependencies required for WinInet.dll (see Table 1 for the API names). You can find these three files in the ActiveX SDK within the Wintdist.exe setup program. (These files are free with MSDN enrollment. See the sidebar, "Redistributing Internet Explorer".)
This is not an "unauthorized hack." Just the opposite: Microsoft has documented the Internet API and provides these files in Wintdist.exe specifically for this purpose.
There are three logical components to communicating with a server over the Internet. First, you need a program that can format and send data to the server: the data source. You can write this program using VB. The actual Internet code is small and is usually just a single module in your program.
Second, you need a program on the server that can read your data, process it, then respond: the data sink. Normally you'd use a CGI program written in Perl or another scripting language, but you can write real CGI programs in VB no clunky controls required. You just need to examine environment variables, and read and write to the server's standard in and out (STDIN and STDOUT) ports using VB and a handful of Windows API calls. Of course, you can't run your VB CGI programs on a Unix server; you need Windows.
Third, and most important, you need a suite of protocols that can handle all the messy details. The protocols are built into IE in the form of WinInet.dll, and its dependency DLLs: InLoader.dll and Schannel.dll. These include Hypertext Transfer Protocol (HTTP) and Hypertext Transfer Protocol Secure (HTTPS). HTTP communicates between a server and a client (normally a Web server and an Internet browser, respectively). It's the lingua franca of the Internet: the protocol that allows your browser to connect to and converse with a server. HTTPS is a secure implementation of HTTP that adds a layer of security to HTTP. With HTTPS, you can use the Secure Sockets Layer (SSL) protocol to cipher all data sent and received to or from the browser and server. Using the IE system files, you can open, read, and write a Transmission Control Protocol/Internet Protocol (TCP/IP) "pipeline" to the server.
Once you establish an Internet pipeline to the server using TCP/IP and HTTP or HTTPS, a data source program can read and write to the pipeline using standard VB file reading and writing code. The data sink program running on the server requires some protocols too. For this article, you'll use the CGI specification and write your server CGI programs using VB6.
Get Connected
This process only works for users who have an Internet ISP and a valid connection or account. It works on any machine with IE 3.1 or later. For machines without IE installed, your users need to use the Wintdist.exe self-installer or the IEAK provided by Microsoft (see the sidebar, "Redistributing Internet Explorer"). Wintdist.exe is small (about 400K) and updates a machine without IE with the required system files. Wintdist.exe won't install IE, nor will it change any system settings with regard to browser settings.
Once Wintdist.exe installs the required system files, the data source program has several requirements: Open the Internet TCP/IP HTTP channel, format data for URL submission (URL encoding), write to the channel using HTTP, read from the channel using HTTP, close the channel, and clean up.
Opening an Internet channel to a server is easy if you can decipher the nearly nonexistent Microsoft documentation. It all boils down to using the WinInet.dll. You can find the WinInet.dll in the System directory of any machine with IE. As mentioned earlier, WinInet.dll is documented on MSDN and other sources, so it's fine to use it for these purposes as long as you follow the Microsoft license.
It helps to think about the Internet channel like a file: You open it, you read it, you write it, and you close it. You use the same technique for opening a channel to your server. In fact, the Internet API declared and exposed by WinInet.dll uses a metaphor similar to opening a file. Take a look at the first API, InternetOpen:
Declare Function InternetOpen Lib "wininet.dll" Alias _
"InternetOpenA"(ByVal sAgent As String, _
ByVal lAccessType As Long, ByVal sProxyName As String, _
ByVal sProxyBypass As String, ByVal lFlags As Long) As Long
The declaration for InternetOpen opens an Internet channel. The arguments to InternetOpen control many aspects of the channel, the key arguments being sAgent and lAccessType. lAccessType controls what kind of generic Internet connection you're trying to set up. lAccessType has defined constants to use with InternetOpen, either with FTP or, in our case, with standard HTTP communications.
Setting lAccessType to the defined INTERNET_ OPEN_TYPE_PRECONFIG constant (decimal 0) tells InternetOpen to use the client machine's existing Internet connection definition. This is important because you won't have to manually set up information about dialers, LANs, modems, firewalls, or proxy servers. It assumes the user has an existing Internet connection, and that the required information is in the Registry.
The system uses the scUserAgent argument to track your sessions. It's a user-defined string, so you can simply use the application name. Here's the code to open an Internet channel:
Const INTERNET_OPEN_TYPE_PRECONFIG = 0
Dim scUserAgent As String
scUserAgent = App.ExeName
hOpen = InternetOpen(scUserAgent, _
INTERNET_OPEN_TYPE_PRECONFIG, "", "", 0)
If you're using a proxy server, then use code such as this:
Const INTERNET_OPEN_TYPE_PRECONFIG = 0
Dim scUserAgent As String
Dim ProxyName As String
scUserAgent = App.ExeName
ProxyName = "your proxy name here"
hOpen = InternetOpen(scUserAgent, _
INTERNET_OPEN_TYPE_PROXY, ProxyName, "", 0)
InternetOpen returns a Long integer value which, if not zero, is the handle for future operations with the channel. Store the result of InternetOpen; you need it for the next step of the process. Also, just like a file, you must close the channel using the InternetCloseHandle API:
Private Declare Function InternetCloseHandle Lib _
"wininet.dll"(ByVal hInet As Long) As Integer
Use CloseHandle to close all opened handles. Closing the handle is easy:
InternetCloseHandle hOpen
Connect to a Server
Some of you might now be asking yourselves, "Hey, where do I define the server I'll connect to?" Indeed, once the channel is open, you need to connect to a server. The server is the HTTP address of the computer you want to connect to. This is commonly a domain. The API for the domain uses this code:
Declare Function InternetConnect _
Lib "wininet.dll" Alias "InternetConnectA"( _
ByVal hInternetSession As Long, _
ByVal sServerName As String, _
ByVal nServerPort As Integer, ByVal sUsername As String, _
ByVal sPassword As String, ByVal lService As Long, _
ByVal lFlags As Long, ByVal lContext As Long) As Long
InternetConnect takes a URL and establishes a session to the server resource specified. Like InternetOpen, InternetConnect takes a few arguments. One of the more bizarre aspects of this API is that you use the INTERNET_INVALID_PORT_NUMBER constant (hey, I didn't name it!) as the server port number to connect to. You pass InternetConnect the channel handle from InternetOpen, a server name, and a port to connect to:
Const INTERNET_INVALID_PORT_NUMBER = 0
Const INTERNET_SERVICE_HTTP = 3
Dim Server As String
Server = "http://www.yourcompany.com"
hConnection = InternetConnect(hOpen, Server, _
INTERNET_INVALID_PORT_NUMBER, _
"", "", INTERNET_SERVICE_HTTP, 0, 0)
Don't specify the CGI program or other resource to access. At this stage of your connection, you're simply opening a channel to communicate with the server. Just like InternetOpen, InternetConnect returns a Long integer, which, if not zero, is a handle to the session opened with the server—so don't lose the hConnection value. All the Internet Open APIs require that you close them and clean up after yourself when you're done.
You still need to identify the specific resource on the server to access. Specify what server resource you want to actually connect to; in this case, a CGI program running on the server:
Declare Function HttpOpenRequest Lib _
"wininet.dll" Alias "HttpOpenRequestA"( _
ByVal hHttpSession As Long, ByVal sVerb As String, _
ByVal sObjectName As String, ByVal sVersion As String, _
ByVal sReferer As String, ByVal lpAny As Long, _
ByVal lFlags As Long, ByVal lContext As Long) As Long
HttpOpenRequest takes many arguments, not all of which apply to what you're doing. See MSDN online for a complete listing of the many arguments, constants, flags, and other functions this API uses (see Links for an index to all the Internet APIs). When working with CGI programs, you only need to worry about sVerb, sObjectName, and lFlags. sVerb is a string indicating what you want to do during the open request. For working with CGI programs, use the string "POST". sObject-Name is a string with the name of the resource on the server you want to access for example, a CGI program name. You can use several verbs, including "POST" and "GET". Believe it or not, you can use "GET" to send and receive data just like "POST". However, it's generally considered a no-no to use "GET", because "GET" has potential security holes, making it much less secure than the "POST" method. Surprisingly, Web applications still use "GET" frequently.
This code connects to a CGI program named sample.exe found in the /cgi-bin directory of the www.yourcompany.com domain (this is a fictitious example; no program is located at this URL):
Const INTERNET_FLAG_KEEP_CONNECTION = 4194304
Dim CGI As String
CGI = "/cgi-bin/sample.exe"
hURL = HttpOpenRequest(hConnection, _
"POST", CGI, "", "", 0, _
INTERNET_FLAG_KEEP_CONNECTION, 0)
If HttpOpenRequest is successful, it returns another Long integer handle with a value greater than zero (yes, save this value, you have to close it too).
Send Data to the Server
The final step is to send some data to the server. This is almost trivial now that you've established the connection. The HttpSendRequest API passes your data to the server, which in turn passes it to the CGI program:
Declare Function HttpSendRequest Lib _
"wininet.dll" Alias _
"HttpSendRequestA" ( _
ByVal hHttpRequest As Long, _
ByVal sHeaders As String, _
ByVal lHeadersLength As Long, _
ByVal sOptional As String, _
ByVal lOptionalLength As Long) As _
Integer
Essentially, HttpSendRequest passes a specially formatted string, containing form data, to the CGI program running on the server over the TCP/IP session established. You can pass optional headers to the server as well. They tell the server what sort of data to expect; for example, sending binary data, multipart forms, or other nontextual information. In this implementation, I use the plain default (HTML) and don't use headers.
HttpSendRequest returns a value not equal to zero if it fails, so code like this is all it takes to send data to a CGI form handler. Call the API, passing your data and the length of the string containing the data. This code uses the m_cPostBuffer variable to handle it:
If HttpSendRequest(hURL, "", 0, _
m_cPostBuffer, Len(m_cPostBuffer)) _
Then
'// it worked!
End If
The process is fairly easy to implement once you figure out the sequence of calls and get the arguments typed correctly. You've now performed almost all the necessary tasks: initialized an Internet TCP/IP channel using InternetOpen, opened a connection to a server using InternetOpen, opened a connection to a resource (CGI program) on the server using InternetConnect, sent the data to the CGI program using HttpSendRe-quest, and closed all the handles using InternetCloseHandle.
I've left out a few important details while giving you the big picture. The most significant detail: The HTML protocol expects its data in a specific format. The m_cPostBuffer variable you used to pass information to a form handler (such as a CGI program) must meet two important criteria.
First, the data is in the form of field_name = field_data. For example, if you have a data field named Company with a value of Your Company, the string needs to be company=your company. Use the ampersand character to concatenate multiple fields. If you have a field titled something like last_name with a value of marquis, the new string would be company=your company&last_name=marquis. Continue this process for all the data you want to pass to the CGI program.
Second, once you build the string, you must use HttpSendRequest to URL-encode it before it's ready to send. The HTTP protocol specifically defines the characters it can and can't transport. You must send encoded strings rather than plain text because, among other things, your data could get corrupted. Instead, encode the data, replacing forbidden characters with a code set composed of allowed characters. Then, at the server end, the CGI program performs a decoding process, exchanging the codes for the forbidden characters. The encoding and decoding process uses simple substitution and only requires a few lines of code to implement.
For example, with company=your company, HTML forbids the space (ASCII 32, hex 20) between your and company. Under the HTML specification, you must replace spaces with a plus (+) character. So, the URL-encoded data used with HttpSendRequest would then be company=your+company&last_name=marquis. (For a complete URL encoding function, see Listing A.) The Url-Encode function in Listing A URL-encodes raw data before you send it to the server. (If you parse and use the data on the server, you might need a complementary UrlDecode routine in your CGI programs.)
Read the Response
If you go through the trouble of opening a channel, formatting, and sending data, it would be nice to read the response from the form handler or CGI program. If you're writing your own CGI form handlers for the server as well, you can pass yourself anything you want. Your CGI program can run calculations, access a database, perform any action you want, then return the results to you as a URL-encoded string.
The API for reading the response from the form handler or CGI program is InternetReadFile:
Private Declare Function _
InternetReadFile _
Lib "wininet.dll" ( _
ByVal hFile As Long, _
ByVal sbuffer As String, _
ByVal lNumBytesToRead As Long, _
lNumberOfBytesRead As Long) As _
Integer
InternetReadFile reads the response from the CGI program just as you would read a file. It reads a fixed number of bytes at a time, and when it finds the "end" of the CGI program response, it concludes. Note the lNumberOfBytesRead argument in the declaration. This isn't a type, and doesn't use ByVal. Unlike most Windows APIs, InternetReadFile returns information to you not from its return result, but into the lNumberOfBytesRead argument directly.
Call InternetReadFile over and over in a loop until it returns no more data, at which point the process is complete. Then, simply return the data collected, which is in HTML format, probably with familiar HTML coding such as <BODY> (see Listing 1). You need to parse the data, bypassing <HEAD>, <p>, and other HTML codes to extract the data you care about. In my implementation, the CGI program returned a string in the body of the HTML code, much like the string passed to the CGI program. My response was fieldname1=fielddata&fieldname2=fielddata, and so on, so I could parse the return easily.
Finally, close the open handles to release the resources used, clean up, and exit the procedure. Add an error handler to close all open handles; unlike VB, WinInet.dll won't clean up for you.
Also note that you can open a connection and keep it open as long as you want. In my case, I wanted a clean, stable "in and out" implementation, so I created the code in Listing A. My PostForm function sends data to a form handler, and returns the result of the post as the function results. (See the sidebar, "Secure Site Tricks," for more details about using secure sites.)
You can write the entire program code for Internet-enabling your programs in a matter of minutes, using a single class module and a few dozen lines of code. You'll find many uses for this process, such as checking for new versions, customer satisfaction, product inquiries, and marketing surveys.
Hank Marquis is the founder and CEO of Modern Software Publishing Inc., which provides next-generation Internet e-commerce technology. Hank was a founding member of the Component Vendor Consortium (CVC) and was chairman of the CVC Component Quality Committee. Hank's new book is A Visual Basic Programmer's Toolkit (APress). Contact Hank at hankm@modernsoftware.com.